Optimizing Time Series Databases for Machine Learning

Articles From: Toggle AI
Website: Toggle AI

Toggle uses AI & Machine Learning to help investors extract insights on their portfolio & investments. The system distills billions of data points into alerts like

“Analyst expectations are turning negative for AAPL, historically this led to stock’s outperformance.” 

As you can imagine, this sort of system requires a tremendous amount of timeseries data — prices, fundamentals, sentiment, etc. All of this data is stored as series and needs to be readily available and accessible for analysis by our models. It is critical that every step in the process is optimized.

When we were first building TOGGLE, we tested many databases, including InfluxDB, Mongo, Cassandra, and TimescaleDB. Initially, we settled on InfluxDB, but as the scale of data grew, performance started to degrade and it became expensive to run. After a short time, we had a small cluster of 4 x m4.2xlarge machines, and memory on all 4 was often at least 80%, hitting 100% a few times per week. 

Knowing all of the data we intended to add (additional fundamentals, positioning & flow data, as well as alt-data like ESG, and geopolitical risk), we knew that we had to find a better solution.

In order to evaluate our options, we created a checklist and knew that the new solution would have to meet every criteria on our list:

  1. Data can be migrated seamlessly & in a timely manner
  2. Response times must have lower latency than before
  3. New data can be ingested seamlessly
  4. New time series can be created on the fly
  5. Performance is maintained once all data has been migrated

We ended up selecting QuestDB – the only database to meet all of the requirements. Below you can see some exciting performance metrics. 

The average time per request is 13 milliseconds, while the system sends nearly 2400 requests per second.

As everyone knows, timeliness in trading is key. This type of improved performance helps us do two things: deliver insights to investors faster, and cover even more data. As mentioned before, we cover more than 800,000 series in order to provide insights on 15,000 assets globally and continue to expand. 

In order to help other investors implement a performant database like this, we are working closely with the QuestDB team to open-source our deployment configuration and migration scripts. Let us know if you’re interested in hearing more! https://www.toggle.global/

Disclosure: Toggle AI

IB Global Investments LLC, a subsidiary of Interactive Broker Group Inc., the parent company of Interactive Brokers LLC, is a minority owner of Toggle AI.

Disclosure: Interactive Brokers

Information posted on IBKR Campus that is provided by third-parties does NOT constitute a recommendation that you should contract for the services of that third party. Third-party participants who contribute to IBKR Campus are independent of Interactive Brokers and Interactive Brokers does not make any representations or warranties concerning the services offered, their past or future performance, or the accuracy of the information provided by the third party. Past performance is no guarantee of future results.

This material is from Toggle AI and is being posted with its permission. The views expressed in this material are solely those of the author and/or Toggle AI and Interactive Brokers is not endorsing or recommending any investment or trading discussed in the material. This material is not and should not be construed as an offer to buy or sell any security. It should not be construed as research or investment advice or a recommendation to buy, sell or hold any security or commodity. This material does not and is not intended to take into account the particular financial conditions, investment objectives or requirements of individual customers. Before acting on this material, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice.