Rest API with a limit of 50 item par pages

Hello,


Our application is using Rest API with a limit of 50 item par pages and there is 2000 pages, so how can I fetch all pages with UrlDownload method GET, or should I use another step?


Best regards


Nicolas

What you are describing is the called Pagination (fetch first 50, fetch next 50 until 2000)...

This can be achieved with the APICall Step which supports pagination.

Thank you for your supports.

I can see that I can use

${nextUrl('http://put.the.next.url.here.com')} fonction like so:

${nextUrl('http://api.myshop.com/getproducts')}

${nextUrl('http://api.myshop.com/getproducts?page=2')}

${nextUrl('http://api.myshop.com/getproducts?page=3')}

${nextUrl('http://api.myshop.com/getproducts?page=4')}

...

but there no automatique way to do this automatically to go to the next page? There is about 240 pages to go and that grow overtime, so it's hard to know exactly how many pages there is exactly, plus, I don't think is a good idea to write 240 lines in parsingTemplate.

Best regards,

Nicolas

It will do this automatically.

The next URLfunction is just needed once. It basically works like that:

The APICall Step calls "itself" again and again if there is a nextUrlFunction. It is like an do-while-loop you know from programming.


We have clarified the explanations in our manual a bit more.


What you need is roughly this:


<#assign row = target.addRow()>
  
<#assign numRows = 0 />
  
<#list json as p>
 <#assign row = target.addRow()>
    ${addColumns(row, p['product'])}
    <#assign numRows = numRows + 1 />
</#list>
  
  
<#if (numRows > 0 && callcounter < callLimit)>
    <#assign nextPageNumber = (callcounter + 1)>
    ${nextUrl(initialUrl+"&page="+nextPageNumber)}
</#if>


The last 4 lines is the important stuff.

It basically is responsible for increasing the page number (page=2, page=3, page=4 etc.)


You may need to tune the stop-if-condition around a little bit too, to avoid an endless loop.
<#if (numRows > 0 && callcounter < callLimit)>


You may need to find out if you already fetched all products. That depends on your API. Some APIs return the total number of pages in the response (e.g. totalPages=10).
(the totalPages value you would have to extract from your API-JSON-Response)


If that is the case you could rewrite it to:


<#if (numRows > 0 && callcounter < totalPagesFromAPIResponse && callcounter < callLimit)>



Ok, thanks it should work now, but I have another problem.


Is VisualJSON2Spreadsheet working after APIcall step? because every time I try to configure VisualJSON2Spreadsheet the screen in jumping back to the project.

No, VisualJson2Spreadsheet is not working, because it expects a FILE as input. But the output of APICall is a Spreadsheet.

The reason is that you need to do the parsing yourself in APICall. That is what the parsingTemplate is for.

Hi,


I have a problems to configure the passingTemplate as you can see on the picture the id doesn't go back to a new row, Everything is in one row. how can I separate those?


image


Here is my parsingTemplate


<#assign row = target.addRow()>

<#list json as j >

<#assign row = target.addRow()>

${addColumns(row, j['entries'])}

</#list>


Thanks


Nicolas

Hello Nicolas,


Looking at the response you're getting this looks like a Plenty Call.

When parsing the template you have define your "loop point".

Let's say your Json looks like this:

"name":"John",
    "age":30,
    "cars": [
        { "name":"Ford", "models":[ "Fiesta", "Focus", "Mustang" ] },
        { "name":"BMW", "models":[ "320", "X3", "X5" ] },
        { "name":"Fiat", "models":[ "500", "Panda" ] }
    ]

You want to make a new row for each car.

This means you have to iterate over the "cars" array.

Your parsing template should probably look like this:

<#assign row = target.addRow()>
<#list json['cars'] as car>
<#assign row = target.addRow()>
    ${addColumns(row, car)}
</#list>

If you want to iterate over each model in each car you have to use 2 <#list>together like this:

<#assign row = target.addRow()>
<#list json['cars'] as car>
<#list car['models'] as model>
<#assign row = target.addRow()>
    ${addColumns(row, model)}
</#list>
</#list>

Sadly I don't know what your response looks like/ which call you made, so I can't help.

Why aren't you using the native Addon in Synesty for Plenty? It's very useful imo.


Greetings

Stefan

If you have some kind of Bundle ROW ID (something that Identifies each Bundle) you could us the Group Function together with the Aggregate Function within the Spreadsheetmapper to archieve the desired outcome.


Best regards,

Stefan

Hello,


Yes, that can be done with Spreadsheetmapper, but the problem with this is that the Spreadsheetmapper need to have a predefined columns to work, and bundles can have 2 or 10 items in it, there is no way to know for sure how many columns to setup, plus that can change over time. That can result some data loss.


So I think the best way to make sure to get all data we need this to be set up in parsingTemplate.


What do you think?


Best regards,

Nicolas

Do you really need each VariationId of a Bundle to be in a seperate Column?
If so:

You could remember the current BundleRowID using the "assign" operator like this for example <#assign currentBundle= o["BundleRowID"]>


When creating the next row within the spreadsheet you put an <#if> like this

<#if o["BundleRowID"]==currentBundle><#assign row = target.addRow()></#if>


This will only create a new row if you are within a new bundle.


Next, when parsing the VariationID you could use a 2nd variable as a counter to dynamically generate ColumnNames.

Hello,


Yes, we really need to have a separate columns for bundles not rows, which is why we do not want to use PlentySearchOrders. The PlentySearchOrders plugins has duplicated numbers in OrderHeadOrderID which will be our KEY for further processing.


UrlDownload work exactly as we want, but he doesn't support pages.


Thanks

Hello Nicolas,


have you tried using the approach I described above?

Here are 3 steps you could take to solve the problem:

1) Only create a new Row if the variation isn't a Bundle_Component (OrderItemsRowID of a Bundle = OrderItemsBundleRowID of a Bundle_Component).

2) Recognize Variations inside the response that belong to the same Bundle

3) Use some sort of counter to dynamically generate Column Names for these Variations (Variation1, Variation2...)


I hope I was able to help.


Greetings

Stefan

Hello,

Thank you for all your help.

I tried without success. It's complicated because the OrderItemsRowID row doesn't exist in the API but the value is there with others values in api entries --> properties in red.

"properties": [

{

"orderId": 566657,

"typeId": 3,

"value": "14"

},

{

"orderId": 566657,

"typeId": 7,

"value": "122200"

},

{

"orderId": 566657,

"typeId": 1,

"value": "3"

},

{

"orderId": 566657,

"typeId": 4,

"value": "fullyPaid"

},

{

"orderId": 566657,

"typeId": 2,

"value": "23"

},

{

"orderId": 566657,

"typeId": 33,

"value": "492269"

}

],

And what happen if there is an order with multi bundles?

I think the best approach with this, is just to expand all columns in one row.

Code exemple:

-----------------

<#assign row = target.addRow()>

<#list json["entries"] as a >

<#assign row = target.addRow()>

${addColumns(row, a, "entries", {"autoExpand":"asColumns"})}

<#list a["properties"] as b >

${addColumns(row, b, "properties", {"autoExpand":"asColumns"})}

</#list>

</#list>

-------------------

With this I just need to find how to remove some columns like entriesrelations, entriesproperties, ... in "${addColumns(row, a, "entries", {"autoExpand":"asColumns"})}"

Thanks

Best regards,

Nicolas