CSV Reader NextToken? Use CSV Reader or Editor?

Intro Background 

CSV General Information: CSVs are only CSV's when they're introduced to the workflow. As they're being processed it's much better to think of them as database lites. The data from the CSV is represented using JSON objects and the information is stored in a way that's much more similar to databases than your average spreadsheet.

CSV Editor vs Reader: You may have already seen the documentation section below comparing our CSV Editor and CSV Reader

Still unsure which is best for you?

Please consider the following as well:

• The CSV Reader is most useful for CSVs that are hundreds to millions of rows long ("very large CSV files"). 
  • As detailed in the docs, the Reader works through pagination with asynchronous queries, so you would need to build in a structure in your workflow to poll the job until your query returns.
    • In order to get that amount of information quickly, the CSV reader can't rely on the line-by-line method the CSV editor uses when retrieving information, so it uses an asynchronous method to retrieve data, in much the same way a database would retrieve information.
• The CSV Editor is typically the simplest to use for CSV files that are not too large
  • You can use the "Get Rows" Operation to read a file without any pagination. 
• The CSV Editor can experience throttling during times of high usage across all users, as explained here

The NextToken

If you choose to use the Reader, this token is used for paginating results, where you would need to continue getting results using the "Get query Results" operation.

You'll need to use the QueryExecutionId and NextToken from the output of your CSV reader as inputs above, and include a failsafe/check that stops the loop when there are no more results to retrieve. We would recommend using callable workflows for this process 

For further information on these types of workflows, please consult our docs, and you can also watch our Tray Academy Live videos below:

• How to  Process large amounts of data: https://www.youtube.com/watch?v=Ob99Ooo60bk
• How to build callable workflows: https://www.youtube.com/watch?v=Qq1eWhVld7s