AEP: Play with schema and dataset

 In this post I will call few of the basic of datasets in AEP. To start with here is the schema I created




This is based on a custom class if Record Type (this class comes with one attribute Identifier shown as locked in the schema).

I made the identifier as primary identity.

Created a dataset with the same name as class.

Data Loading

1. Delimitted File

Created a simple delimitted file as below and created a workflow to load it into the dataset
Sample Data:
id,dataset_name,last_snapshot_id,process_timestamp,process_status,failure_reason
journey_step_events_27878,journey_step_events,27878,2024-09-12T19:19:50.036Z,SUCCESSFUL,Not applicable
aa_stitched_events_23451,aa_stitched_events,23451,2024-09-17T19:19:50.036Z,FAILED,There was no snapshot available
journey_step_events_27891,journey_step_events,27891,2024-09-16T19:19:50.036Z,In Process,Not applicable

2. Using JSON

Download the JSON format from the schma page and create JSON file using the format. For multiple records it needs to be array as below. 
[{
    "_mytechnologys": {
        "dataset_name": "aa_stitched_events",
        "failure_reason": "Not applicable JSON",
        "last_snapshot_id": 23049,
        "process_status": "WIP",
        "process_timestamp": "2018-11-12T20:20:39+00:00"
    },
    "_id": "aa_stitched_events_23047"
}]

3. Using SQL

You can insert the record using SQL as well. Here is the sample SQL I was using for this schema
INSERT INTO tg_checkpoint_log
SELECT
  'journey_step_events_27880' as _id,struct('journey_step_events' AS dataset_name,
  27880 as last_snapshot_id,cast(CURRENT_TIMESTAMP AS TIMESTAMP) as process_timestamp, 'WIP' as process_status) as _mytechnologys;

Note: the step and 2 can also be done using API. For file loading, it is recommended to use Parquet file. I didn't try to load csv file using API though.

Here the data is always inserted. Even after making the _id field as primary identifier, while inserting with same values, it does not update records. If you select, you will find duplciate records in the schema.

Comments

Post a Comment

Popular posts from this blog

Avoid Proxy for HttpClientRequest - IOB-090007 Network error (send(), errno=10054: an existing connection was forcibly closed by the remote host

This is Friday night 8:10 PM and I was frustratingly stuck with an annoying issue on something that I have done so many times, but still unable to figure out what is wrong since last night. Finally it was a silly mistake and I found that. So wanted to capture it so that I don't need to spend another day and half just to figure out why the same thing is not working I was making a call to Silverpop XMLAPI to get the jessionid via the login method. And the code is something like below, very basic one var http = new HttpClientRequest ( "http://api2.ibmmarketingcloud.com/XMLAPI" ); try { http . method = 'POST' ; http . header [ "ClientId_Internal" ] = "XYZABCD" ; http . header [ "ClientSecret_Internal" ] = "DSFDSOFNJSDF" ; http . header [ "Content-Type" ] = "application/x-www-form-urlencoded" ; var reqestBody = "xml=" + encodeURIComponent ( "<
Read more

Base 64 Encoding using Adobe Campaign Standard

 For one of the integration we had to do the base64 encoding using ACS and it was not that easy initially with the existence of the "MYTH" that ACS doesn't support JavaScript. Here is the code where I declared a class Converted inside a Content Block (it has to be block, not fragment) named "cbBase64EncodeLib" Note this process is only applicable if you have to use the encoded output inside email, it will not work if you need to save the value in a table <% var Converter ={         _keyStr : "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=",         hexToBase64: function(hex) {             //var str = Converter.hexToString(hex);             //return Converter.base64Encode(str);             var hexStr = hex.toString(); //force conversion             var binary = "";             for (var i = 0; i < hexStr.length; i++) {                 var h = hexStr.charAt(i);                 temp = parseInt(h, 16).toString(2);      
Read more

Campaign classic sequence of execution of different Javascript Code Bloc inside a workflow delivery

Image
 Every time I have to add some advanced JS code inside Delivery I am confused about the sequence of JS. When it comes to Delivery, then there are multiple places where you can write the JS  Right Click on Delivery > Open > Go to Script Tab  Right Click on Delivery > Open > Go to Advanced Tab Typology Rule of type Control There are many phases where the rule will be run. I was trying to run the rule as much late as possible and thus had chosen "At the end of the analysis" Ignoring the option of adding JS code inside the Routing External account > Connector and Post processing workflow as I didn't run that Here is the result While the delivery is waiting at "Approve Targeting" , the log for the recurring delivery activity at workflow level looks like below The delivery log looks like below for the Typology Rules Note: At this stage the broadlog is not created and that makes sense as the Targeting is not approved yet While the  delivery is waiting at
Read more