Mapping and Modifiers

The Mapping node is used to link data from a Source system to a Destination system. The data that is being mapped is formatted using Modifiers.

Introduction

Mapping in DCKAP Integrator is used to link data from a source system to a destination system. Data from the source fields are formatted using Modifiers and then mapped to the associated destination fields.

Integrations or Pipes are the components that process the data synchronization based on the associated Flows and Mapping configurations. Based on the selected credentials, every pipe is associated with one Flow. The Flow may/may not contain one or more Mapping nodes that are added while developing the Flow.

More on Integrations can be found here.

More on Flows can be found here.

Concept of Mapping

Flows are the neural schema of every integration. A flow determines how the data should be synched end to end. A flow usually contains one or more Mapping nodes, which will format and link data from the source system to the destination system.

In the Flow, the API calls to be used in the mapping node are defined.

These API calls are configured in the API manager either used to get data or post data. Also, sample response and request fields are added while configuring these APIs.

More on API Manager can be found here.

The fields from the sample 'Request' and 'Response' are then used in the Mapping nodes as source and destination fields respectively.

Creating a New Pipe

1. Open the Integrations tab and click on Add New.

2. In the Add Pipe page, enter a Pipe Name and select Credentials (systems).

3. Once the Credentials have been chosen, all the Flows associated with those systems will be listed in the drop-down menu. Choose a Flow.

4. After a flow has been selected, all the Mapping nodes defined in the Flow will be displayed as Mapping cards.

5. Click on Configure to make changes to the Mapping nodes.

Mapping Configuration

Click on Configure in the Mapping Card and a new window will open where you can

• Link fields

• Format data

• Modify data using modifiers

• Add custom fields

• Preview the mapping configuration

• Import or export mapping configurations

Here you will see Source fields on the Left-Hand side and Destination fields on the Right-Hand side. These fields are extracted from the sample Request and Response sections in the API Manager.

How is Mapping done?

Drag and drop fields onto the Mapping area in the center.

For example:

• first_name, last_name from the source mapped to customer_name, name in the destination

• email in the source to email_address in the destination.

Drag and Drop feature in Mapping

Add Custom Field

The Add Custom Field is used to add a new data value to the Source Field which can then be modified and sent to the destination.

Remember to hit the Save button after you have used the Modifier feature.

Modifier functions

Fields for which Modifiers have been applied in the Play area, cannot be directly removed from the Mapping page.

If any field from that is used in the Play area is removed from the API manager response, then that corresponding field (row) wil be highlighted in the Mapping play area.

Comment Mapping

A specific mapping configuration can be temporarily disabled by using the Comment option.

Preview Mapping

The Preview Mapping is used to test and get a preliminary view of how the Mapping functionality will work, instead of testing the entire Integration, thereby saving the developer's time.

Click on the Preview button in the top left section of the Mapping page. The Preview Mapping window will open.

1. Source Data: In this section, add the Source field data in JSON format.

2. Ignore Keys: Keys that should be ignored by the system, while processing data in the mapping structure defined by the user.

Click on the Preview button. Depending on the Source data provided and the Mapping and Modifier configurations, the preview response will display the output in the format of the respective destination fields.

Save Options

• Save & Continue: To save the Mapping and remain on the same page

• Save & Exit: To save the Mapping and return to the Pipe page

Use Case

SOURCE DATA
{"firstname":"John", "lastname":"Smith"}

IGNORE KEYS: Customer

PREVIEW RESPONSE
{
	"address": {
		"name": "JohnSmith"
	},
	"company_id": "CLO",
	"contact": {
		"first_name": "John",
		"last_name": "Smith"
	},
	"customer_name": "JohnSmith"
}

Modifier Functions

Modifiers may be added to each field that is mapped or to the link that connects two fields. The Modifier is used to edit or format data from the source fields.

Add one or more fields into each line of the Mapping Area. After adding fields to the Mapping area, click on the Modifier icon , in the middle. Here, the fields are modified and transformed to a format that satisfies the underlying Mapping logic.

How to configure Modifiers?

1. Drag one or more Source fields and drop it to the Play Area.

2. Then one or more Modifier functions maybe added to the source fields and also the link. Select the field and then Drag and Drop the modifier functions to the Configuration area or Play Area.

3. The Group feature is used to group one or more fields. The Grouped entity can then be used as a single field for modification.

4. Use the Preview section to test the Modifier functionality. The input format of the data to be modified can be selected by clicking on the drop down list in the Preview section.

Example: firstname (concatenate) lastname. The output of this modifier is mapped to the destination fields.

BUILT IN FUNCTIONS

int

bit_length

Returns the number of bits necessary to represent an integer in binary, excluding the sign and leading zeros.

conjugate

Returns the complex conjugate of any int.

Example:
1+3j.conjugate()
Result: (1-3j)

list

Append

Adds an object to the end of the list.

Syntax: l.append(object)
Parameters: element (required)

Example: 
sample_array = [a,b,c]
element = d
sample_array.append(element)

Result: [a,b,c,d]

count

Returns the number of occurrences of a value in the list.

Syntax: l.count(value)

Example:
sample_array = [a,b,c,a]
sample_array.count(a) 

Result: 2

extend

Extends list by appending elements from the iterable.

Syntax: l.extend(iterable)
Parameters: array_value (required)

Example: 
sample_array1 = [a,b,c]
sample_array2 = [d,e,f]
sample_array1.extend(sample_array2) 

Result: [a,b,c,d,e,f]

index

Returns the index of the first occurrence of the specified list item. Raises ValueError if the value is not present.

Syntax: l.index(value, [start, [stop]])

Parameters:
end
index (required)
start

Example #1:
sample_array = [a,b,c,a]
sample_array.index(a,1,5)
Result: 3

Example #2: 
sample_array = [a,b,c,a]
sample_array.index(a)
Result: 0

insert

Inserts an object at the specified index position.

Syntax: l.insert(index, object)

Parameters:
element (required)
index (required)

Example: 
sample_array = [a,b,c,a]
sample_array.insert(2,f)

Result: [a,b,f,c,a]

pop

Removes and returns an item at the specified index (The default is to return the last item in the list). "IndexError" error is raised if list is empty or index is out of range.

Example: 
sample_array = [a,b,c,a]
sample_array.pop(2)
Result: [a,b,a]

remove

Remove first occurrence of value. Raises ValueError if the value is not present.

Syntax: l.remove(value)
Parameters: element

Example: 
sample_array = [a,b,c,a]
sample_array.remove(c)
Result: [a,b,a]

reverse

Reverse *IN PLACE*

Syntax: l.reverse()

sample_array = [a,b,c,a]
sample_array.reverse ()
Result: [a,c,b,a]

sort

Stable sort *IN PLACE*

Syntax: l.sort(key=None, reverse=False)

Parameters:
compare
key
reverse

Str

capitalize

Return a capitalized version of the string, i.e. make the first character have upper case and the rest lower case.

Syntax: S.capitalize() -> str

Example:
sample_string = "test"

Result: Test

center

Return S centered in a string of length width. Padding is done using the specified fill character (default is a space).

Syntax: S.center(width[, fillchar]) -> str

Parameters:
fillchar
width (required)

Example:
sample_string = "test"
fillchar = -
width = 4
Result : ----test----

count

Returns the number of non-overlapping occurrences of substring sub in string.

Syntax: S[start:end]

Optional arguments start and end are interpreted as in slice notation
S.count(sub[, start[, end]]) -> int

Parameters:
char (required)
end
Start

Example:
count 
sample_string = "test"
char = "t"
Result : 2

encode

Encode S using the codec registered for encoding. Default encoding is 'utf-8'.

Errors may be given to set a different error handling scheme. The default value is 'strict' meaning that encoding errors raise a UnicodeEncodeError.

Other possible values are 'ignore', 'replace' and 'xmlcharrefreplace' as well as any other name registered with codecs.register_error that can handle UnicodeEncodeErrors.

Syntax: S.encode(encoding, errors)

Example: S.encode(encoding='utf-8', errors='strict') -> bytes

endswith

Returns True if the String ends with the specified suffix; False otherwise. With optional start, tests the String from the beginning at that position. With optional end, stop comparing S at that position. Suffix can also be a tuple of strings to try.

Syntax: S.endswith(suffix[, start[, end]]) -> bool
Parameters:
end (required)
start (required)
suffix (required)

Example:
sample_string = "test"
end 
start: starting and ending index
suffix: st
Result: True

expandtabs

Return a copy of S where all tab characters are expanded using spaces. If tabsize is not given, a tab size of 8 characters is assumed.

Syntax: S.expandtabs(tabsize=8) -> str
Parameters: tabsize

find

Return the lowest index in S where substring sub is found, such that sub is contained within S[start:end]. Optional arguments start and end are interpreted as in slice notation. Return -1 on failure.

Syntax: S.find(sub[, start[, end]]) -> int

Parameters:
char (required)
end
start

Example:
sample_string = "test"
char = "e"
Result: 1

index

Return the lowest index in S where substring sub is found, such that sub is contained within S[start:end]. Optional arguments start and end are interpreted as in slice notation. Raises ValueError when the substring is not found.

Syntax: S.index(sub[, start[, end]]) -> int

Parameters:
char (required)
end
start

Example:
sample_string = "test"
char = "e"
Result: 1

isalnum

Return True if all characters in S are alphanumeric and there is at least one character in S, False otherwise.

Syntax: S.isalnum() -> bool

Example:
sample_string = "test12"
Result: True 

isalpha

Return True if all characters in S are alphabetic and there is at least one character in S, False otherwise.

Syntax: S.isalpha() -> bool

Example:
sample_string = "test"
Result: True 

isdigit

Return True if all characters in S are digits and there is at least one character in S, False otherwise.

Syntax: S.isdigit() -> bool

Example:
sample_string = "123"
Result: True

islower

Return True if all cased characters in S are lowercase and there is at least one cased character in S, False otherwise.

Syntax: S.islower() -> bool

Example: 
sample_string = "test"
Result: True

isspace

Return True if all characters in S are whitespace and there is at least one character in S, False otherwise.

Syntax: S.isspace() -> bool

Example:
sample_string = "   "
Result: True

istitle

Return True if S is a titlecased string and there is at least one character in S, i.e. upper- and titlecase characters may only follow uncased characters and lowercase characters only cased ones. Return False otherwise.

Syntax: S.istitle() -> bool

Example:
sample_string = "Test"
Result: True

isupper

Return True if all cased characters in S are uppercase and there is at least one cased character in S, False otherwise.

Syntax: S.isupper() -> bool

Example:
sample_string = "TEST"
Result: True

join

Return a string which is the concatenation of the strings in the iterable. The separator between elements is S.

Syntax: S.join(iterable) -> str

Parameters: array_values (required)

Example:
sample_list = ["test", "one", "try"]
sample_string = "-"
sample_string.join(sample_list)
Result= "test-one-try"

ljust

Return S left-justified in a Unicode string of length width. Padding is done using the specified fill character (default is a space).

Syntax: S.ljust(width[, fillchar]) -> str

Parameters:
fillchar
width (required)

Example: 
sample_string = "test"
fillchar = -
width = 6
Result: test--

lower

Return a copy of the string S converted to lowercase.

Syntax: S.lower() -> str

Example:
sample_string = "TEST"
Result: "test"

lstrip

Return a copy of the string S with leading whitespace removed. If chars is given and not None, remove characters in chars instead.

Syntax: S.lstrip([chars]) -> str
Parameters: prefix

Example:
sample_string = "  test"
Result: "test"

partition

Search for the separator sep in S, and return the part before it, the separator itself, and the part after it. If the separator is not found, return S and two empty strings.

Syntax: S.partition(sep) -> (head, sep, tail)
Parameters: separation (required)

Example: 
sample_string = "test"
sample_string.partition("s")
result: ("te", "s", "t")

replace

Return a copy of S with all occurrences of substring old replaced by new. If the optional argument count is given, only the first count occurrences are replaced.

Syntax: S.replace(old, new[, count]) -> str
Parameters:
count
new (required)
old (required)

Example: 
sample_string = "test"
new = w
old = t
Result = "west"

rfind

Return the highest index in S where substring sub is found, such that sub is contained within S[start:end]. Optional arguments start and end are interpreted as in slice notation. Return -1 on failure.

Syntax: S.rfind(sub[, start[, end]]) -> int
Parameters:
char (required)
end
Start

Example:
sample_string = "test"
char = "t"
Result: 3

rindex

Return the highest index in S where substring sub is found, such that sub is contained within S[start:end]. Optional arguments start and end are interpreted as in slice notation. Raises ValueError when the substring is not found.

Syntax: S.rindex(sub[, start[, end]]) -> int

Parameters:
char (required)
end
start

Example:
sample_string = "test"
char="t"
Result: 3

rjust

Return S right-justified in a string of length width. Padding is done using the specified fill character (default is a space).

Syntax: S.rjust(width[, fillchar]) -> str

Parameters: fillchar
width (required)

Example: 
sample_string = "test"
fillchar = -
width = 6
Result: --test

rpartition

Search for the separator sep in S, starting at the end of S, and return the part before it, the separator itself, and the part after it. If the separator is not found, return two empty strings and S.

Syntax: S.rpartition(sep) -> (head, sep, tail)
Parameters: separation (required)

Example:
sample_string = "test"
sample_string.rpartition("t")
result: ("tes", "t", "")

rsplit

Return a list of the words in S, using sep as the delimiter string, starting at the end of the string and working to the front. If maxsplit is given, at most maxsplit splits are done. If sep is not specified, any whitespace string is a separator.

Syntax: S.rsplit(sep=None, maxsplit=-1) -> list of strings

Parameters:
char (required)
maxsplit

Example:
sample_string = "test-try"
char= "-"
result: ["try","test"]

rstrip

Return a copy of the string S with trailing whitespace removed. If chars is given and not None, remove characters in chars instead.

Syntax: S.rstrip([chars]) -> str
Parameters: chars

Example:
sample_string = "test  "
Result: "test"

split

Return a list of the words in S, using sep as the delimiter string. If maxsplit is given, at most maxsplit splits are done. If sep is not specified or is None, any whitespace string is a separator and empty strings are removed from the result.

Syntax: S.split(sep=None, maxsplit=-1) -> list of strings

Parameters:
char (required)
maxsplit

Example:
sample_string = "test-try"
char= "-"
result: ["test","try"]

startswith

Return True if S starts with the specified prefix, False otherwise. With optional start, test S beginning at that position. With optional end, stop comparing S at that position. prefix can also be a tuple of strings to try.

Syntax: S.startswith(prefix[, start[, end]]) -> bool

Parameters:
end
prefix (required)
Start

Example:
sample_string = "test"
end 
start: starting and ending index
suffix: te
Result: True

strip

Return a copy of the string S with leading and trailing whitespace removed. If chars is given and not None, remove characters in chars instead.

Syntax: S.strip([chars]) -> str

Example:
sample_string = "  test  "
Result: "test"

swapcase

Return a copy of S with uppercase characters converted to lowercase and vice versa.

Syntax: S.swapcase() -> str

Example:
sample_string = "TeSt"
Result: "tEsT"

title

Return a titlecased version of S, i.e. words start with title case characters, all remaining cased characters have lower case.

Syntax: S.title() -> str

Example:
sample_string = "test"
Result: "Test"

upper

Return a copy of S converted to uppercase.

Syntax: S.upper() -> str

Example:
sample_string = "test"
Result: "TEST"

zfill

Pad a numeric string S with zeros on the left, to fill a field of the specified width. The string S is never truncated.

Syntax: S.zfill(width) -> str
Parameters: count (required)

Example:
sample_string = "test"
width=10
Result:"test000000"

Converter

converter

To Modify the data with a static raw data from the user input

Eg:- Original data {country:"India"}

Requested Modification {country:"IN"} :returns: Modified JSON data

The Swap option will allow users to interchange the Key and Value data.

Example: 

key="India"
value="IN"

Then IN will be sent to the destination in place of India

Example of Default value:

key= "__default__"

value= "US"

If the key is an empty key or any key other than IN it will take the default value "US".

CUSTOM

1. 12Hr to 24Hr Converter and 24Hr to 12Hr

Converts Time from one format to the other (12Hr clock to 24Hr clock and vice versa).

Format to enter data
12Hr: 3:00PM
24Hr: 15:00

2. Array Extender

Used to Concatenate two or more strings and then convert the output of the concatenation into a single string.

Applicable Only To Lists.

Example: 
first_name=John
last_name=Smith
Marks=[23, 45, 54, 10, 34]

Result: [John,Smith,23,45,54,10,34]

3. Array or String Slicing

To slice values from the string/array

Parameters:
start_index
step
stop_index

Example 1: 
start = 0 or None,
stop = 2, 
step = 1 
sample = "slicingoperation" 
Returns "sl" 


Example 2: 
start = -4, 
stop = None, 
step = 1 
sample = "slicingoperation" 
Returns "tion"


Example 3: 
start = None, 
stop=None, 
step=-1 
sample = "slicingoperation" 
Returns "noitarepognicils" (Reversed Value)

All the above examples applies to arrays as well:

• param start_index: Index of the element from where you want to start the slicing operation

• param stop_index: Index of the element where you want to stop the slicing operation

• param step: Specifies the steps that you want to make during the slicing operation

4. Array To String

To convert any datatype to string if its dict. Applies json dumps if its list, set, tuple. Joins the iterable elements using join function.

Parameters: join_characters

Example:
sample_array = [a,b,c,d]
join_characters = ","
Result: "a,b,c,d"

5. Concatenate

Used to Concatenate two or more strings and, convert the output of the concatenation into a single string.

Parameters: join_chars

Example: 
str1 = "test" 
str2 = "try" 
Result: "testtry"

6. Convert Non-ASCII characters to ASCII Characters

Example:
Input1 : char_a = "Škoda"
Output1 : Skoda (Converted string from Non-ASCII to ASCII)

7. Converter

To Modify the data with a static raw data from the user input

Eg:- Original data {country:"India"}

Requested Modification {country:"IN"} :returns: Modified JSON data

The Swap option will allow users to interchange the Key and Value data.

The Converter modifier in the CUSTOM modifier section can be used in conjunction with other Modifiers.

Example: 

key="India"
value="IN"

Then IN will be sent to the destination in place of India

Example of Default value:

key= "__default__"

value= "US"

If the key is an empty key or any key other than IN it will take the default value "US".

8. Current Datetime

Used to display the current date and time in the project timezone.

9. Datatype Converter

To convert the the source value to requested datatype

1. string

2. integer

3. dict (requires tuple of key and value pairs)

4. list

5. float

Parameters: Datatype (required)

Example:
sample_value = "1"
Datatype = "integer"
Result: 1 (converted string to integer)

10. DateTime Formatter

To convert the date_value to the requested format from current format.

Parameters: 
date_value: Date String
requested_format (required): Date format of the return value

Example:
date_value = 2020-05-18 06:25:03.737Z
requested_format = YYYY-MM-DD 
Result: 2020-05-18

Complete list of format codes:

Format Codes	Description	Example
%d	Day of the month as a zero-padded decimal number	01, 02, 03, 04 …, 31
%a	Weekday as abbreviated name	Sun, Mon, …, Sat
%A	Weekday as full name	Sunday, Monday, …, Saturday
%m	Month as a zero-padded decimal number	01, 02, 03, 04 …, 12
%b	Month as abbreviated name	Jan, Feb, …, Dec
%B	Month as full name	January, February, …, December
%y	Year without century as a zero-padded decimal number	00, 01, …, 99
%Y	Year with century as a decimal number	0001, …, 2018, …, 9999
%H	Hour (24-hour clock) as a zero-padded decimal number	01, 02, 03, 04 …, 23
%M	Minute as a zero-padded decimal number	01, 02, 03, 04 …, 59
%S	Second as a zero-padded decimal number	01, 02, 03, 04 …, 59
%f	Microsecond as a decimal number, zero-padded on the left	000000, 000001, …, 999999
%I	Hour (12-hour clock) as a zero-padded decimal number	01, 02, 03, 04 …, 12
%p	Locale’s equivalent of either AM or PM	AM , PM
%j	Day of the year as a zero-padded decimal number	01, 02, 03, 04 …, 366

11. Datetime to Timestamp Modifier

This Modifier is used to provide the timestamp starting from <MM/DD/YYYY> to a given Date & Time input.

• Give the input as a string

• In the Parameters field, pass the string format using the table

• Provide the timezone of the user input

The supported timezone parameters are listed here: Supported Timezones

12. Email Validator

To validate the email address. If valid, return the email address, else return the default value.

Parameters: default

Example: 
sample_email= [e-mail]

Default email: [e-mail]

13. Expression

Based on the logical operations that the user requests, the resulting data will be calculated.

Parameters: logical (required)

Example 1: 
firstName = "integrator", 
lastName="India" 
expression: and 
results: integratorIndia 

Example 2: 
firstName = "integrator", 
lastName="India" 
expression: or 
results: integrator

14. Find and Match Key value

To convert any datatype to string if its dict. Applies json dumps if its list, set, tuple. Joins the iterable elements using join function.

Parameters: 
Check_value (required)
default (to set a default value if the check_value 
does not match any of the array elements)
Find_key (required)
Result_key (required)

Example:
input_data=
[
  {
    "attribute_code":"EA",
    "value":2
  },
  {
    "attribute_code":"USA",
    "value":3
  },
  {
    "attribute_code":"IN",
    "value":4
  }
]

check_value = "EA"
find_key = "attribute_code"
Result_key = "value"
Result:2

15. Get Index Value

To get the index value from string or list datatype.

Parameters:
default_value
index (required)

Example: 
sample_string = "test"
index = 2
Result: "s"

default: "r"
index = 7
Result: "r"

16. Group

Grouping will apply precedence over the grouped elements.

Example:
(firstName or lastName) and company 
(integrator or app) and DCKAP

Result: integratorDCKAP

17. Mathematical Operations

To perform Arithmetic operations such as:

• Addition

• Subtraction

• Multiplication

• Division

• Floor division

• Power of

• Modulo operation (remainder)

18. Multiple String to Array

Used to push two or more string into an array.

str1 = "test", 
str2 = "try" 
returns ["test","try"] 

19. Number from String Modifier

This method is used to retrieve numbers from the address string.

Parameters:

• index: Index of the number to be retrieved

Example
Input: "14/3, 3055 GRAND AVE 75215, DALLAS, Texas"  
 
Case 1:
    If index is not set,
    Result= "14/3"
    
    Explanation:
    Here the numbers from the address are: ['14/3', '3055', '75215'].
    By default, index 0 will be returned if the index value is not set.

Case 2:
    If index is set to 2,
    Result= "75215"  

20. Push Into an Array

To push elements into a new array.

Example:
sample_value = 
{
    "attribute_code":"EA",
    "value":2
}

Result: [{
    "attribute_code":"EA",
    "value":2
}]

21. Raw Value

The value assigned to Raw Value will be taken into consideration.

Parameters: value (Required)

consider_source = true : to check if there is data in the source 
consider_source = false : will consider the raw data

default of consider_source = false

22. RegEx

A RegEx, or Regular Expression, is a sequence of characters that forms a search pattern.

RegEx can be used to check if a string contains the specified search pattern.

23. Remove Non-ASCII characters from the input data

Example:

Input2 : char_b = "Škoda"
Output2 : koda (Removed Non-ASCII character from the string.)

24. Remove Special Characters

To remove special characters from a string.

Parameters:
exceptions - Provide special characters that must not be removed from the string
selections - Provide special characters that must be removed from the string

Sample_string="te$t"
Result:"tet"

Sample_string="te$t?"
exceptions=?
Result:"tet?"

sample_string="te$t?"
selections=?
Result:"te$t"

Provide either selections or exceptions. If both the selections and exceptions are provided, selections will be used.

25. Round Off

To round off to the nearest integer value

Parameters: 
digits : Number of decimals to use when rounding

Example:
sample value = 3.7
Result = 4

26. Truncate

Removes the specific number of characters in the prefix or suffix of a string based on the parameters provided. By default, characters from the right of the string are removed.

Parameters: 
count - Defines the number of characters to be removed
reverse - Set to True by default, (True = characters will be removed from the right)

Example: 
string = "test" 
count = 2 
results = "te"

27. 12Hr to 24Hr Converter and 24Hr to 12Hr

Converts Time from one format to the other (12Hr clock to 24Hr clock and vice versa).

Format to enter data
12Hr: 3:00PM
24Hr: 15:00