PokeAPI - Demonstrating Pagination, Recursive Loops, and Variable XPath Queries

I needed a token-less API to play with and figure out pagination. Google Books API was a bit boring and wanted something a bit more interesting. Enter PokeAPI.

There weren’t too many posts here that fully discussed and resolved pagination, especially with breakers (when it is unknown when the last “next link” will be provided), so after resolving this, I thought to share a complete, working solution and demonstrative workflow to paginate an API and concatenate results.

Overview:

The workflow performs the following key sets of activities:

  1. Gets lists of Pokemon and the link to the next list

  2. Loops over the links and table headers, collecting results each time

  3. Breaks the recursive loop once the next link is missing

Final table with all 1302 Pokemon:

This result table could be taken a step further and call each Pokemon ID (with the captured links) to further query their stats, etc.

Key Learning:

  • The Loop Breaker is powered by the Rule Engine node, and that node specifically has to out put “true”: in lowercase, and captured in quotes.

  • Subsequently, the Table Row to variable immediately following the Rule Engine node has to output “false” in the default case (i.e., next link still present). Otherwise, a blank string will fail (the use of variable to end recursive loop depends on explicitly seeing either “true” or “false”)

  • Care should be taken in the XPath node querying header values: this should output “Multiple Rows” and not Single Cell. This is an easy mistake to miss, as your final table will still look like a proper table with many rows - but in fact your total row count is not the rows of the data available, but the number of iterations that were run. The PokeAPI is helpful here as we all know how many Pokemon there are.

Applications:

  • Paginating APIs with unknown page numbers

  • Finding XPath headers when the path is known, but attributes are many

Hope this was helpful. Happy to get feedback as well on where this could get improved. I could see potential opportunities in either optimizing the way XPath attribute-headers are grabbed (which can be slow when there are hundreds of attributes and thousands of records per page), as well as consolidating the two GET Request nodes, which in turn would eliminate the need for downstream concatenation.