Tracking Token Usage in AI Workflow Nodes

Overview, Architecture & Flow

When running AI workflows that use large language models or other API-driven AI services, tracking token consumption is essential for managing costs and performance. The Azure AI workflow nodes that Softools offer in workflows support token usage tracking through integration with an AI Token Tracking App.

This configuration allows you to:

• Record token usage per workflow run

• Aggregate token consumption per API key

• Monitor usage trends over time

• Receive alerts for unusually high usage

In order to set this up we require using the following approach.

Note: We have built and deployed the app that is showcased in this article. If you would like to use our App as a base then feel free to reach out to support@softools.net and we can support you in getting it copied across to your site.

1. Build the AI Tracking App 

The AI Tracking App needs the following Fields to be able to accurately report how many tokens have been used and when:

• Project: This is a custom Field that allow you to group workflow nodes to track how many tokens are being used for what purpose. This options could include Insights, Competitor Analysis & Process Automation,
• App: Having a Field that states which App the Workflow Node runs in allow for quick analysis of which App is using what proportion of Tokens
• Run ID: It is best practice to have a unique reference for each Record in an App but beyond this we can insert the request ID that is returned in the meta data when a Workflow Node runs. This means we can do some investigation if the tracking App pulls back any surprise results.
• API Key: Using separate keys per site or project allows you to track usage better and analyse where costs are incurred.
• Tokens Used: This is the key value in this App that state the number of tokens that have been used in each individual workflow node run.
• Time Stamp: With the time the token usage happened, we can then start to aggregate token usage by day, week, month or as needed for token usage alerts.

For additional tracking you may also want to know who triggered the workflow. In this case it will often be that the workflow is set up to be actioned by the person who triggered it. If this is true then you can add a Triggered By Field with the expression as follows:

• Triggered By: CreatedBy()

   

To begin with add a Table listing the workflow node runs with these key Fields. We will return to the Tracking App Set Up later to look at how we set up alert rules and cost/usage insight.

2. Configuring the AI Workflow Nodes

For complete instructions on configuring an AI workflow Node please consult the Send Actions (Notifications, Webhooks, Emails & AI Nodes) support article.

Self Managed AI Nodes (Require own API access) - For AI tracking, the Azure OpenAI Prompt and Azure OpenAI Classification nodes require you to use your own API Key which means you can give them a friendly reference in the tracking App if using multiple keys.

Managed AI Nodes (API Infrastructure built-in) - The Web Search node and Send Managed AI prompt node are offered courtesy of a Softools' API Key with inbuilt shared limits.

Once the workflow has run it will need to use values returned in the meta data of the AI workflow node in order to create the tracking record. Let's take an example where we click a button to run AI Insight in an Inspection App.

The structure will need five nodes:

First Node: Trigger the AI Workflow Node

For this example I have chosen the simplest node which is that the AI node will run on the click of a button. See here for a full list of Workflow Triggers.

Second Node: Run the AI Workflow Node

This is where we will use one of the AI Workflow Nodes. Where relevant, be sure to set the API Key and the Token limit per run of the workflow node.

Third Node: Update Field Value with Meta Data Values

The third node is what to do with the result of the AI node. In our use case we have added a Field Updated node to paste the value that we get back from the AI Web Search into a Field in the Inspections App.

Fourth Node: Update Field Value with Meta Data Values

The fourth node is setting some variables which we can then use to create a record in our AI tracking App. The important part here for tracking AI is that the following two variables are created in our workflow that will return the headers and full content of the payload in response from the AI workflow node.

• ResponseHeaders: [Meta.ResponseHeaders]

• ResponseContent: [Meta.ResponseContent]

Fifth Node: Create an AI Tracking Record

This node is the Create Record action and it will sends the usage data to the AI Token Tracking App. The Project, App and API Key values are entered as static text values as appropriate for classifying the AI run and the RunID, Tokens Used and Time Stamp record the usage stats that we pulled out of the meta data from the AI workflow node in node three of this flow.

• Project: 'AI Insights'

• App: 'Inspections'

• RunID: jsonValue(sourcevar('ResponseHeaders','{}'), 'X-Request-ID', '')

• APIKey: 'API-Key-001'

• TokensUsed: jsonValue(sourcevar('ResponseContent','{}'), 'usage.total_tokens', 0)

• TimeStamp: dateParse(jsonValue(sourcevar('ResponseHeaders','{}'), 'Date', ''))

If you would like to break down the AI tokens further you could also add Fields to your AI Tracking App for Prompt Tokens and Completion Tokens which breaks down the Total AI Tokens used into these two parts.

These can be obtained from the ResponseContent variable using:

• PromptTokens: jsonValue(sourcevar('ResponseContent','{}'), 'usage.prompt_tokens', 0)

• CompletionTokens: jsonValue(sourcevar('ResponseContent','{}'), 'usage.completion_tokens', 0)

Another piece of information which may be of use is to know the AI Model used. This is particularly useful when different models have different pricing and you would like to calculate billing.

• AIModel: jsonValue(sourcevar('ResponseContent','{}'), 'model', '')

There are more custom Fields that can be added but the last one that we would recommend is to record the Record that triggered the Workflow. To do this requires two steps:

• Additional Variable to Node 4: Add a variable for TriggerRecord with expression id()

• Use Variable in an additional Field: Add a Field for Trigger Record in your tracking App and then in the workflow add this new Field to this Record Create Node with expression sourcevar('TriggerRecord','')

3. Meaningful Reports on AI Usage

Now that we are tracking the AI Token Usage per Workflow Run, we can aggregate this to give meaningful insight into how AI tokens are being used in workflow Nodes.

AI Usage Splits

One way to do this is to understand where there is most usage of Tokens across API Key, Project or App. By adding a cumulative token count pie chart for each of these, we can then add the below Dashboard which shows how usage is distributed and hence where most investment is being utilised. Knowing this allows for appropriate review on AI token usage strategies. These are configured as Pie Charts.

Periodic Usage

The other important usage statistics to surface is the amount of AI tokens used over which time frames. Below is a report that shows how much usage there has been by month.

To set these Reports up we create a Cumulative Matrix Report. The grouping Field for the X-Axis doe require a specific Field to be set up in the App. These have the following expressions:

• Month (Period Field): if(isValidDate([TimeStamp]), string(year([TimeStamp])) + '-' + if(month([TimeStamp]) < 10, '0', '') + string(month([TimeStamp])) + '-01T00:00:00Z','') 
• Week (Text Field): if(isValidDate([TimeStamp]), string(year([TimeStamp])) + '-' + if(weekNum([TimeStamp]) < 10, '0', '') + string(weekNum([TimeStamp])), '')

4. Monitoring and Alerts

Regularly checking the AI Tracking Dashboards and Reports will enable you to keep on top of how many tokens are being used and where. It will also enable you to see how usage scales over time. What is important though is not to be surprised by increased usage.

This is done by setting up workflows that act as an alert message such as:

• Alert: An API Workflow Node Run has hit the max token limit.

• Alert: API Token Usage has exceeded 20,000 tokens in October 2025.
  Recommendation: Review workflow configuration or increase token quota.

Single API Run has hit max token limit

For this we can pull back two additional pieces of information from the AI Workflow Node ResponseHeaders.

First add fields in the AI Token Tracker App for Remaining Requests and Remaining Tokens. Then we can update our workflow that creates the AI token Tracking Record to include the following two values

• Remaining Requests: jsonValue(sourcevar('ResponseHeaders','{}'), 'x-ratelimit-remaining-requests', '')
• Remaining Tokens: jsonValue(sourcevar('ResponseHeaders','{}'), 'x-ratelimit-remaining-tokens', '')

These are rate limits so they will reset but we are recording them to show if a rate limit has been hit by an AI workflow run.

We can now add a Single Run Rate Limit Bit Field that is set to true if either limit is hit:

• SingleRunRateLimitHit: if([RemainingRequests] = 0 || [RemainingTokens] = 0, true, false)

Now we can set up an alert flag workflow that if Single Run Rate Limit Hit is true on Record Create then send an email to the appropriate people who need to review and see if the limits need increasing or the AI usage optimised.

Total API Usage Tokens exceeded monthly limit

To be able to set up this alert we need to know for each record how many AI tokens were already used this month and then what the total is including this run. This way we can set up an alert trigger to be when the AI Token count exceeds the monthly limit. So to do this you can add the following two Fields:

Prior Monthly Token Usage:
aggregate('sum','false','5fcf4ba9c783c7147dfd814b','AITokenTracker','TokensUsed','{"Month": ISODate("' + string(year([TimeStamp])) + '-' + if(month([TimeStamp]) < 10, '0', '') + string(month([TimeStamp])) + '-01T00:00:00Z"), "TimeStamp": { $lt: ISODate("' + string(year([TimeStamp])) + '-' + if(month([TimeStamp]) < 10, '0', '') + string(month([TimeStamp])) + '-' + if(day([TimeStamp]) < 10, '0', '') + string(day([TimeStamp])) + 'T' + if(hour([TimeStamp]) < 10, '0', '') + string(hour([TimeStamp])) + ':' + if(minute([TimeStamp]) < 10, '0', '') + string(minute([TimeStamp])) + ':' + if(second([TimeStamp]) < 10, '0', '') + string(second([TimeStamp])) + 'Z") }}')

Total Monthly Token Usage:
aggregate('sum','false','5fcf4ba9c783c7147dfd814b','AITokenTracker','TokensUsed','{"Month": ISODate("' + string(year([TimeStamp])) + '-' + if(month([TimeStamp]) < 10, '0', '') + string(month([TimeStamp])) + '-01T00:00:00Z"), "TimeStamp": { $lt: ISODate("' + string(year([TimeStamp])) + '-' + if(month([TimeStamp]) < 10, '0', '') + string(month([TimeStamp])) + '-' + if(day([TimeStamp]) < 10, '0', '') + string(day([TimeStamp])) + 'T' + if(hour([TimeStamp]) < 10, '0', '') + string(hour([TimeStamp])) + ':' + if(minute([TimeStamp]) < 10, '0', '') + string(minute([TimeStamp])) + ':' + if(second([TimeStamp]) < 10, '0', '') + string(second([TimeStamp])) + 'Z") }}') + [TokensUsed]

Also, we will want to know which Record triggered the AI Usage Quota alert so we add a Monthly Quota Limit Exceeded bit Field which we can set to true.

Having these three Fields we can now add an alert email to go out for the AI Workflow Run that hits the Monthly Quota. This is a workflow on record create trigger in the AI Token Tracking App.

• Node 1: Add a Record created Trigger
• Node 2: Add a filter with the API Alert Quota so that prior use was below the quota and total usage now at the quota -- [TotalTokensThisMonthPrevious] < 20000 && [TotalTokensThisMonth] >= 20000
• Node 3:  Add an Update Field Value action node that flags the alert bit field to true
• Node 4: Add a Send Email action node to alert the appropriate users that a quota limit has been reached.

The example flow in the screenshot demonstrates how multiple alerts can be set. When building these limits think about whether you would like to be alerted by total usage or by API Key / App / Project... This will require more fields in the Token Tracker App to total up usage by the desired segmentation.

Best Practices

• Set conservative max_tokens values to prevent cost overruns.

• Regularly review usage reports to identify high-consumption workflows.

• Use API key segmentation — separate keys per project or environment.

• Enable alerting early to avoid unexpected billing spikes.