AEP: Play with schema and dataset

 In this post I will call few of the basic of datasets in AEP. To start with here is the schema I created




This is based on a custom class if Record Type (this class comes with one attribute Identifier shown as locked in the schema).

I made the identifier as primary identity.

Created a dataset with the same name as class.

Data Loading

1. Delimitted File

Created a simple delimitted file as below and created a workflow to load it into the dataset
Sample Data:
id,dataset_name,last_snapshot_id,process_timestamp,process_status,failure_reason
journey_step_events_27878,journey_step_events,27878,2024-09-12T19:19:50.036Z,SUCCESSFUL,Not applicable
aa_stitched_events_23451,aa_stitched_events,23451,2024-09-17T19:19:50.036Z,FAILED,There was no snapshot available
journey_step_events_27891,journey_step_events,27891,2024-09-16T19:19:50.036Z,In Process,Not applicable

2. Using JSON

Download the JSON format from the schma page and create JSON file using the format. For multiple records it needs to be array as below. 
[{
    "_mytechnologys": {
        "dataset_name": "aa_stitched_events",
        "failure_reason": "Not applicable JSON",
        "last_snapshot_id": 23049,
        "process_status": "WIP",
        "process_timestamp": "2018-11-12T20:20:39+00:00"
    },
    "_id": "aa_stitched_events_23047"
}]

3. Using SQL

You can insert the record using SQL as well. Here is the sample SQL I was using for this schema
INSERT INTO tg_checkpoint_log
SELECT
  'journey_step_events_27880' as _id,struct('journey_step_events' AS dataset_name,
  27880 as last_snapshot_id,cast(CURRENT_TIMESTAMP AS TIMESTAMP) as process_timestamp, 'WIP' as process_status) as _mytechnologys;

Note: the step and 2 can also be done using API. For file loading, it is recommended to use Parquet file. I didn't try to load csv file using API though.

Here the data is always inserted. Even after making the _id field as primary identifier, while inserting with same values, it does not update records. If you select, you will find duplciate records in the schema.

Comments

Post a Comment

Popular posts from this blog

Avoid Proxy for HttpClientRequest - IOB-090007 Network error (send(), errno=10054: an existing connection was forcibly closed by the remote host

This is Friday night 8:10 PM and I was frustratingly stuck with an annoying issue on something that I have done so many times, but still unable to figure out what is wrong since last night. Finally it was a silly mistake and I found that. So wanted to capture it so that I don't need to spend another day and half just to figure out why the same thing is not working I was making a call to Silverpop XMLAPI to get the jessionid via the login method. And the code is something like below, very basic one var http = new HttpClientRequest ( "http://api2.ibmmarketingcloud.com/XMLAPI" ); try { http . method = 'POST' ; http . header [ "ClientId_Internal" ] = "XYZABCD" ; http . header [ "ClientSecret_Internal" ] = "DSFDSOFNJSDF" ; http . header [ "Content-Type" ] = "application/x-www-form-urlencoded" ; var reqestBody = "xml=" + encodeURIComponent ( "...
Read more

Looping using subset inside a big batch in Adobe Campaign

Image
 I had many scenarios where I ran a costly query to select some good no of records and then process them further. But when it comes to processing I had to restrict the number of records to be processed in one go. And once the subset is processed, then select from the table again and then process the subset. Something like below where the end activity calls the signal when the complement from Split has more than 0 records to start the whole process again. But the problem is every time you have to run the costly select process to process a fraction of the records. As a result the overall throughput is decreased heavily. As an alternate I used a looping through the selected records avoiding selection every time I need to process the subset. Here the JS Activity after select is dummy one but required. I added a log in it with count. Test activity is also needed to ensure we have the exit path from the loop. In the above example the select query fetched 8680 records and then split restr...
Read more

Displaying data in an email from one to many relation table

Image
We know that boradLogRcp has a one to many relationship (1 to N) with recipient schema. So for every recipient I want to show all the emails sent to him/her Create a simple workflow to select the recipients of your choice Add a delivery/recurring delivery and pull the OTB default email template and configure it with below code <TABLE> <TBODY> <TR> <TD>Campaign name</TD> <TD>Delivery Name</TD> <TD>Event Date</TD> </TR> <% for(var i = 0 ; i<recipient.broadLog.length; i++) { document.write("<tr>"); document.write("<td>" + recipient.broadLog[i].delivery.operation.label + "</td>"); document.write("<td>" + recipient.broadLog[i].delivery.label + "</td>"); document.writ...
Read more