5 actionable tips for reducing your AI token cost

The costs of conversational & agentic AI are steadily increasing. But at the same time, AI is becoming embedded in more and more business systems. Combined, this makes AI an increasingly prominent line item on your Balance Sheet each quarter.

In this article, I'll explain a simple premise for reducing your AI token usage by increasing the quality of your input.

Although this is a painfully simple premise, bear with me because implementing long-term and repeatable strategies for reducing your token usage can be difficult. In this article, I'll outline the methods we (and our customers) use to consistently reduce their costs:

1. Reducing your context size
2. Use tools to defer context load
3. Teach an AI to fish
4. Reduce your retries
5. Pre-calculate responses

Reducing your context size

When you initiate a chat with your AI, it doesn't actually begin with the first question you ask. Before this, your AI provider (ourselves included) will preload something called "context" into your chat. This context is how we prepare the AI to help you best (in our case, it is to help you with your business data) and it can be a significant contributor to the size of your AI payload. Let's demonstrate with a real example from SYNCHUB, where we're interrogating our Shopify data. Here is the conversation as you (the human) might see it, starting with the question from yourself (1):

Forgetting about the AI's reply (for now), we can see this is just six words - a negligible cost for your AI. But here's the same chat from the AI's point of view:

Note that in actuality, the user's question is the second input into the chat. The first (and at this point, most important) input is the context we have loaded in about a) the Shopify data; and b) how the AI can extract information about this data (more on this later). This context is approximately 500 words (ie. an 83× increase in token usage.

So wait - I thought this was about reducing costs?

Although this context significantly increases your AI costs, let's pause for a moment and consider the additional value that this has added to your conversation. Previously, your AI knew nothing about your business at all. Now, our context has provided the AI with an in-depth knowledge of your product, inventory, sales & customer databases (ie. your Shopify data). Without this context, your AI is about as useful as asking your cat for business advice - and (more importantly) this context is something that you would have to feed to your AI eventually anyway.

So all else being equal, a thorough and in-depth context is just table-stakes when it comes to querying your business data. But as you'll see below, SYNCHUB's usage is actually significantly more efficient, due to our prudent use of supporting tooling.

Use tools to defer context load

Look closely again at the context fed into our conversation above - in particular, the section outlining the query_data tool towards the bottom. This is an example of how your AI should defer context load wherever possible:

Why should you defer data? Well, look how large the response is when the AI executes the tool:

{
  "DatastoreSchema": "{\"FriendlyName\":\"Shopify\",\"Description\":\"e-commerce platform for creating and managing online stores\",\"DatabaseTypeName\":\"SQLAzure\",\"Tables\":{\"ProductTag\":{\"TableName\":\"ProductTag\",\"SyncEntityTypeID\":355,\"Notes\":[\"Always return the Tag column in your query results\"],\"Columns\":{\"ProductRemoteID\":{\"ColumnType\":\"nvarchar(max) NULL\",\"ForeignKeyTo\":\"Product.RemoteID\",\"Notes\":[]},\"ProductID\":{\"ColumnType\":\"bigint NOT NULL\",\"ForeignKeyTo\":\"Product.ID\",\"Notes\":[]}}},\"Fulfillment\":{\"TableName\":\"Fulfillment\",\"SyncEntityTypeID\":358,\"Columns\":{\"OrderRemoteID\":{\"ColumnType\":\"nvarchar(max) NULL\",\"ForeignKeyTo\":\"Order.RemoteID\",\"Notes\":[]},\"OrderID\":{\"ColumnType\":\"bigint NULL\",\"ForeignKeyTo\":\"Order.ID\",\"Notes\":[]},\"LocationID\":{\"ColumnType\":\"bigint NULL\",\"ForeignKeyTo\":\"Location.ID\",\"Notes\":[]}}},\"OrderLineItem\":{\"TableName\":\"OrderLineItem\",\"SyncEntityTypeID\":357,\"Columns\":{\"OrderRemoteID\":{\"ColumnType\":\"nvarchar(max) NULL\",\"ForeignKeyTo\":\"Order.RemoteID\",\"Notes\":[]},\"OrderID\":{\"ColumnType\":\"bigint NULL\",\"ForeignKeyTo\":\"Order.ID\",\"Notes\":[]},\"VariantID\":{\"ColumnType\":\"bigint NULL\",\"ForeignKeyTo\":\"ProductVariant.ID\",\"Notes\":[]},\"ProductID\":{\"ColumnType\":\"bigint NULL\",\"ForeignKeyTo\":\"Product.ID\",\"Notes\":[]}}},\"ProductOption\":{\"TableName\":\"ProductOption\",\"SyncEntityTypeID\":354,\"Columns\":{\"ProductRemoteID\":{\"ColumnType\":\"nvarchar(max) NULL\",\"ForeignKeyTo\":\"Product.RemoteID\",\"Notes\":[]},\"ProductID\":{\"ColumnType\":\"bigint NOT NULL\",\"ForeignKeyTo\":\"Product.ID\",\"Notes\":[]}}},\"AbandonedCheckoutShippingLine\":{\"TableName\":\"AbandonedCheckoutShippingLine\",\"SyncEntityTypeID\":3453,\"Columns\":{\"AbandonedCheckoutRemoteID\":{\"ColumnType\":\"nvarchar(max) NULL\",\"ForeignKeyTo\":\"AbandonedCheckout.RemoteID\",\"Notes\":[]},\"AbandonedCheckoutID\":{\"ColumnType\":\"bigint NOT NULL\",\"ForeignKeyTo\":\"AbandonedCheckout.ID\",\"Notes\":[]}}},\"FulfillmentLineItem\":{\"TableName\":\"FulfillmentLineItem\",\"SyncEntityTypeID\":359,\"Columns\":{\"FulfillmentRemoteID\":{\"ColumnType\":\"nvarchar(max) NULL\",\"ForeignKeyTo\":\"Fulfillment.RemoteID\",\"Notes\":[]},\"ID\":{\"ColumnType\":\"bigint NULL\",\"ForeignKeyTo\":\"OrderLineItem.ID\",\"Notes\":[]},\"FulfillmentID\":{\"ColumnType\":\"bigint NULL\",\"ForeignKeyTo\":\"Fulfillment.ID\",\"Notes\":[]},\"OrderID\":{\"ColumnType\":\"bigint NULL\",\"ForeignKeyTo\":\"Order.ID\",\"Notes\":[]}}},\"AbandonedCheckoutShippingLineTaxLine\":{\"TableName\":\"AbandonedCheckoutShippingLineTaxLine\",\"SyncEntityTypeID\":3456,\"Columns\":{\"AbandonedCheckoutShippingLineRemoteID\":{\"ColumnType\":\"nvarchar(max) NULL\",\"ForeignKeyTo\":\"AbandonedCheckoutShippingLine.RemoteID\",\"Notes\":[]},\"ShippingLineRemoteID\":{\"ColumnType\":\"nvarchar(max) NULL\",\"ForeignKeyTo\":\"AbandonedCheckoutShippingLine.RemoteID\",\"Notes\":[]}}},\"AbandonedCheckoutLineItem\":{\"TableName\":\"AbandonedCheckoutLineItem\",\"SyncEntityTypeID\":3454,\"Columns\":{\"AbandonedCheckoutRemoteID\":{\"ColumnType\":\"nvarchar(max) NULL\",\"ForeignKeyTo\":\"AbandonedCheckout.RemoteID\",\"Notes\":[]},\"AbandonedCheckoutID\":{\"ColumnType\":\"bigint NOT NULL\",\"ForeignKeyTo\":\"AbandonedCheckout.ID\",\"Notes\":[]},\"ProductID\":{\"ColumnType\":\"bigint NULL\",\"ForeignKeyTo\":\"Product.ID\",\"Notes\":[]},\"VariantID\":{\"ColumnType\":\"bigint NULL\",\"ForeignKeyTo\":\"ProductVariant.ID\",\"Notes\":[]}}},\"ProductVariant\":{\"TableName\":\"ProductVariant\",\"SyncEntityTypeID\":353,\"Columns\":{\"ProductRemoteID\":{\"ColumnType\":\"nvarchar(max) NULL\",\"ForeignKeyTo\":\"Product.RemoteID\",\"Notes\":[]},\"ProductID\":{\"ColumnType\":\"bigint NOT NULL\",\"ForeignKeyTo\":\"Product.ID\",\"Notes\":[]},\"InventoryItemID\":{\"ColumnType\":\"bigint NULL\",\"ForeignKeyTo\":\"InventoryItem.ID\",\"Notes\":[]}}},\"OrderLineItemTaxLine\":{\"TableName\":\"OrderLineItemTaxLine\",\"SyncEntityTypeID\":1146,\"Columns\":{\"OrderLineItemRemoteID\":{\"ColumnType\":\"nvarchar(max) NULL\",\"ForeignKeyTo\":\"OrderLineItem.RemoteID\",\"Notes\":[]},\"OrderLineItemID\":{\"ColumnType\":\"bigint NOT NULL\",\"ForeignKeyTo\":\"OrderLineItem.ID\",\"Notes\":[]}}},\"OrderTaxLine\":{\"TableName\":\"OrderTaxLine\",\"SyncEntityTypeID\":1145,\"Columns\":{\"OrderRemoteID\":{\"ColumnType\":\"nvarchar(max) NULL\",\"ForeignKeyTo\":\"Order.RemoteID\",\"Notes\":[]},\"OrderID\":{\"ColumnType\":\"bigint NOT NULL\",\"ForeignKeyTo\":\"Order.ID\",\"Notes\":[]}}},\"AbandonedCheckoutTaxLine\":{\"TableName\":\"AbandonedCheckoutTaxLine\",\"SyncEntityTypeID\":3455,\"Columns\":{\"AbandonedCheckoutRemoteID\":{\"ColumnType\":\"nvarchar(max) NULL\",\"ForeignKeyTo\":\"AbandonedCheckout.RemoteID\",\"Notes\":[]},\"AbandonedCheckoutID\":{\"ColumnType\":\"bigint NOT NULL\",\"ForeignKeyTo\":\"AbandonedCheckout.ID\",\"Notes\":[]}}},\"AbandonedCheckoutNoteAttribute\":{\"TableName\":\"AbandonedCheckoutNoteAttribute\",\"SyncEntityTypeID\":3452,\"Columns\":{\"AbandonedCheckoutRemoteID\":{\"ColumnType\":\"nvarchar(max) NULL\",\"ForeignKeyTo\":\"AbandonedCheckout.RemoteID\",\"Notes\":[]},\"AbandonedCheckoutID\":{\"ColumnType\":\"bigint NOT NULL\",\"ForeignKeyTo\":\"AbandonedCheckout.ID\",\"Notes\":[]}}},\"RefundLineItem\":{\"TableName\":\"RefundLineItem\",\"SyncEntityTypeID\":827,\"Columns\":{\"RefundRemoteID\":{\"ColumnType\":\"nvarchar(max) NULL\",\"ForeignKeyTo\":\"Refund.RemoteID\",\"Notes\":[]},\"RefundID\":{\"ColumnType\":\"bigint NOT NULL\",\"ForeignKeyTo\":\"Refund.ID\",\"Notes\":[]},\"LineItemID\":{\"ColumnType\":\"bigint NULL\",\"ForeignKeyTo\":\"OrderLineItem.ID\",\"Notes\":[]},\"LocationID\":{\"ColumnType\":\"bigint NULL\",\"ForeignKeyTo\":\"Location.ID\",\"Notes\":[]}}},\"Customer\":{\"TableName\":\"Customer\",\"SyncEntityTypeID\":349,\"Notes\":[\"Always return the FirstName and LastName columns in your query results\"],\"Columns\":{\"DefaultAddressID\":{\"ColumnType\":\"bigint NULL\",\"ForeignKeyTo\":\"CustomerAddress.ID\",\"Notes\":[]}}},\"InventoryLevel\":{\"TableName\":\"InventoryLevel\",\"SyncEntityTypeID\":350,\"Columns\":{\"InventoryItemID\":{\"ColumnType\":\"bigint NOT NULL\",\"ForeignKeyTo\":\"InventoryItem.ID\",\"Notes\":[]},\"LocationID\":{\"ColumnType\":\"bigint NOT NULL\",\"ForeignKeyTo\":\"Location.ID\",\"Notes\":[]}}},\"Location\":{\"TableName\":\"Location\",\"SyncEntityTypeID\":351,\"Notes\":[\"Always return the Name column in your query results\"],\"Columns\":{}},\"Product\":{\"TableName\":\"Product\",\"SyncEntityTypeID\":352,\"Notes\":[\"Always return the Title column in your query results\"],\"Columns\":{}},\"Order\":{\"TableName\":\"Order\",\"SyncEntityTypeID\":356,\"Notes\":[\"Always return the Reference column in your query results\"],\"Columns\":{}},\"SmartCollection\":{\"TableName\":\"SmartCollection\",\"SyncEntityTypeID\":360,\"Notes\":[\"Always return the Title column in your query results\"],\"Columns\":{}},\"Collection\":{\"TableName\":\"Collection\",\"SyncEntityTypeID\":362,\"Notes\":[\"Always return the Title column in your query results\"],\"Columns\":{}},\"Shop\":{\"TableName\":\"Shop\",\"SyncEntityTypeID\":364,\"Notes\":[\"Always return the Name column in your query results\"],\"Columns\":{}},\"AbandonedCheckout\":{\"TableName\":\"AbandonedCheckout\",\"SyncEntityTypeID\":3451,\"Columns\":{\"CustomerID\":{\"ColumnType\":\"bigint NULL\",\"ForeignKeyTo\":\"Customer.ID\",\"Notes\":[]}}}},\"APITypeID\":13}",
  "ProducedByActionID": "1098"
}

Pretty big, right? A lot of AI platforms will naively load this type of context in at the start of a conversation, which means you are paying for the token usage even if you don't use the information!

By using tools effectively, you can guide your AI to load in context only as it is required, and thereby ensuring you're only paying for tokens essential to your current line of questions.

Teach an AI to fish

I'm sure you've heard of that old adage, Give a man a fish, feed him for a day. Teach a man to fish, feed him for a lifetime? It's exactly the same with your AI. Let's consider the problem of data-extraction above and the two mechanisms you could employ to answer the simple question, "What is my top selling product?"

Option one - give a man a fish

Obviously, your AI cannot answer this question if it does not have the raw sales data, and the easiest way to do this is dump a CSV file of your sales into your initial prompt. In terms of output quality, this is excellent and the AI will (almost certainly) return the correct result. But that CSV file is ultimately tokens that you're sending back & forth to your AI, and this costs money - a lot of money. Even an extremely simple & contrived example of a cafe that sold 10 coffees/hour for 40 hours a week would still have something like this...

1 Feb 1979 10am, Flat white, 1

...repeated over 20,000 times, just to upload a single year's worth of data. And here's what's worse - most of that data isn't required, but you're still paying your AI to process it.

Option two - teach a man to fish

If you extend the example above to incorporate all your business data, you can quickly understand that not only is the above unaffordable, it is literally impossible to pre-load your AI with an entire business-worth of knowledge.

But AI is only as good as the data it is fed on, so how do providers get around this limitation? Well, for starters we build tooling at a higher level, helping the AI understand the structure of your data and how it can be accessed. In the case of SYNCHUB specifically, this means that we give your AI the database model (tables, columns, foreigh keys and custom annotations) and instructions on how to query that data.

Next time you find yourself copy/pasting context into your AI prompts, consider if you could actually "teach your AI to fish" instead. The cost savings are potentially enormous.

Reducing retries with improved accuracy

Here's an important thing about your AI conversations that isn't immediately apparent - every message that you send compounds on all prior messages. For example:

Look how your token usage is increasing exponentially, because each time you ask a question, the AI posts back the entire conversation history. This is not a design flaw - this is literally how AI chatbots are designed to work.

What this means for you is that every clarification you make costs you money. But remember, you can also mitigate the need for clarifications by making sure your AI is as smart as possible. At SYNCHUB, this is done with the aforementioned data context, but in your case it may be something as simple as crafting your prompts more thoroughly:

• Anticipate the errors that the AI might make, and address them early
• Provide all the data that your AI needs up-front - don't wait for it to ask
• Being clear and unambiguous in your prompting - despite its familiarity and confidence, your AI knows nothing about you, so speak to it as if you are complete strangers

Pre-calculate your AI responses yourself

This sounds counter-intuitive, but hear me out. Many questions that you ask your AI are actually questions that you've asked before - the question remains the same, but the surrounding variables are different. For example:

In all of these examples, the question remains the same which means that you effectively already know the question ahead of time. And this means that if you can address the variables that impact the answer, then you can likely provide most of the response ahead-of-time and feed it back to the AI yourself.

For example, Claude allows you to share a folder with arbitrary files in it. If you made sure your recent sales were always in this file (e.g. in the form of a CSV file), then Claude wouldn't need to make expensive calculations itself - it just has to load the file contents.

At SYNCHUB we address this with our Insights functionality. Insights allow you to precompute arbitrary (and often extremely long-running) SQL queries and stash their results in an AI-accessible database table. This gives your AI instant access access to aggregates or other computed metrics which might otherwise involving it trawling through sometimes millions of database records to find the result itself.