The ability to extract and analyze historical data efficiently is critical for the success of any quantitative strategy. Thankfully, TWS API affords its users the ability to extract historical data at scale for a number of products including derivatives such as options and futures. While there are many good resources available online explaining how to extract historical data for stocks, I was unable to find any good resource for derivatives. Since many of my students regularly reach out to me seeking help with the same, I have decided to write this two-part article delving into this topic.
While the approach to extract historical data for derivatives and stocks is similar, there are some derivative specific peculiarities which could make things tricky. In addition, getting historical data for expired derivatives contracts is a challenge because IB does not provide historical data for expired options contracts and provides historical data for expired futures contracts only going back a few years. However, there is a way to extract historical data for expired contracts as well. Don’t believe me? Refer to the below screenshot wherein I have extracted the data for an option contract which expired a few months back.
But before you get too excited about raiding the historical data for expired options, let me warn you that the approach has a learning curve and therefore it is important that we go step by step. By the end of this two-part article you should become reasonably proficient in extracting historical data for both live and expired derivatives contracts. In this first part, let us take care of the basics required to extract derivatives data using TWS API.
This article assumes that you have basic understanding about TWS API. If not, please refer to my earlier article about implementing market scanner using TWS API where I have explained the basics of TWS API and the building blocks of a TWS API based trading application.
Building the Application
Before we create a trading application that requests historical data for the given derivative contract, let us build our boilerplate trading application.
As explained in my previous article, when executed, this trading application will simply create a websocket connection with your locally running TWS software. However, since we have not requested any information or sent any commands to the TWS in the above code, this application will not do anything much.
To request historical data from the TWS, we need to execute the reqHistoricalData() Eclient function. We need to pass some arguments to the reqHistoricalData() function so that TWS understands the scope of our request. We will get into the details of this function in a bit but as of now it is important to note that one of the most critical arguments to be passed to this function is the contract for which you want to extract data.
Quick Introduction to IBAPI Contract Class
IBAPI’s contract class lets us define the contract to be used in our trading application. Once we define the contract, we can perform operations such as requesting historical data, submitting orders etc. for that contract. To create an object of the contract class we need to define the relevant parameters that helps TWS identify the exact contract. For example, the critical parameters required to identify a stock are the stock symbol, the security type, the exchange where it is listed and the currency in which it trades. However, the above parameters are not sufficient for derivatives since each derivative contract has an expiry date and multiple derivative contracts for a given underlying may be trading simultaneously. Therefore, it is important that we create our contract by defining the required parameters. TWS API documentation has a very useful section wherein sample contract definition for a number of products is provided https://interactivebrokers.github.io/tws-api/basic_contracts.html and it may be a good idea to refer to this resource.
Now how do we get information on all the parameters required to define a contract? There are a couple of ways to do that. We can add the contract in the TWS monitor panel and then select “description” by right clicking on the contract. This will provide all the details pertaining to the contract. Refer to the below screenshots.
We can also access contract details by searching for the contract in the IB Contract & Symbol Database (https://www.interactivebrokers.com/en/index.php?f=463)
It is important to note that we need to purchase the relevant market data pack from IB in order to access historical data. Thankfully, the subscription fee tends to be quite reasonable and you can subscribe to whichever market data you are interested in from your IB account settings>>User Settings>>Market Data Subscription. While I am not subscribed to the futures market data from the Globex exchange, I do have subscription to the Futures data from India’s National Stock Exchange and that is what I will be using for this demo. Let us now improve our trading application by creating the contract for which we want to extract historical data.
As you can see in the above code snippet, I first imported the Contract class from IBAPI which allows me create objects of the Contract class. I then created an object of the Contract class and proceeded to define the parameters that would help TWS identify the exact futures contract that I am interested in. Now that we have taken care of the most important argument of the reqHistoricalData() let us get into the details of this function.
Quick Introduction to the Historical Data API
TWS API’s reqHistoricalData function provides historical data in the form of successive “bars”. Each bar (or candle) contains the “Open”, “High”, “Low”, “Close” and “Volume” information for a specific time duration in the past. Refer to the the documentation on historical data https://interactivebrokers.github.io/tws-api/historical_bars.html which should provide a good overview of the historical data API.
Let us now expand our trading application code further and add the reqHistoricalData() function.
The parameters for this function are:
reqID – This is the request number which helps TWS identify one request from another
contract – This is the object of Contract class for which historical data is requested
endDateTime – This is the time (in YYYYMMDD format) until which historical data is required. Empty string passed as an argument is interpreted as request to provide data till the current moment.
durationStr – The duration or length or time for which data is required (Refer to historical data documentation for valid values)
barSizeSetting – The granularity of data requested (Refer to historical data documentation for valid values)
whatToShow – The method of creating bar data from bid and trade data e.g. bars created using bid and ask, mid-point, adjusted last price etc. (Refer to historical data documentation for valid values)
UseRTH – Whether you want data within regular trading hours (1) or want pre and post market data as well (0)
fotmatDate – The date format to be returned (use 1)
keepUpToDate – Whether you want unfinished real-time bars (1) or not (0).
chartOptions – Documentation does not provide any detail on this parameter so safe to pass blank list.
Below is our updated code:
As you can see In the above code, we have applied the reqHistoricalData() function to our trading application and passed the relevant arguments to request 30 minute bars for the last 300 days of the future contract of Infosys expiring on 27th Jan 2022.
Now you may ask, where have we defined this reqHistoricalData function? Well, this function is already defined in the EClient class and when we created our trading application (TradeApp), we inherited from the EClient and the Ewrapper classes and therefore have access to all the functions already defined in these classes. The relevant EClient and EWrapper functions for various applications (such as requesting historical data, streaming market data, placing orders etc.) can be obtained from the TWS API documentation (http://interactivebrokers.github.io/tws-api/index.html)
In the above code, we have used the historicalData() Ewrapper function in our trading application. This function handles the response from TWS once we execute the app.reqHistoricalData() function. The TWS returns the reqID and bar information which can be manipulated in the historicalData() function. In the above code, I am simply printing out the received data but you can store the data in your desired data structure (topic for later). Below is the partial output that we receive once we run the updated code.
Quick Introduction to the Historical Data API
Let us now change our contract object and try to extract historical data for an expired future contract. Let me change the lastTradeDateOrContractMonth attribute of the contract object to a date in the past.
Below is the output when we run our updated code.
As expected, TWS does not identify the expired Future contract. Thankfully, all we need to do here is add another attribute, includeExpired, to our contract object which will instruct TWS not to be facetious and provide the data for the expired contract. Please note that IB keeps data of expired contracts going back only a few years (I think it is two years but you may want to double check).
Below is the partial output with the updated code.
Summary & Next Steps
In this article we gained an understanding of the TWS API tools required to extract historical data. We also gained proficiency in creating and tweaking the contract object and learnt how to extract historical data for both live and expired futures contracts. In the follow up article, we will discuss historical data for options which has some additional challenges.
The author can be reached at firstname.lastname@example.org
If you found this article interesting, you may consider enrolling in my Interactive Brokers API centric courses on https://learning.rasuquant.com. The courses range from basics of TWS API to advanced concepts that will help you backtest and build fully automated trading applications on Interactive Broker’s platform.
Disclosure: Interactive Brokers
Information posted on IBKR Traders’ Insight that is provided by third-parties and not by Interactive Brokers does NOT constitute a recommendation by Interactive Brokers that you should contract for the services of that third party. Third-party participants who contribute to IBKR Traders’ Insight are independent of Interactive Brokers and Interactive Brokers does not make any representations or warranties concerning the services offered, their past or future performance, or the accuracy of the information provided by the third party. Past performance is no guarantee of future results.
This material is from Rasuquant and is being posted with permission from Rasuquant. The views expressed in this material are solely those of the author and/or Rasuquant and IBKR is not endorsing or recommending any investment or trading discussed in the material. This material is not and should not be construed as an offer to sell or the solicitation of an offer to buy any security. To the extent that this material discusses general market activity, industry or sector trends or other broad based economic or political conditions, it should not be construed as research or investment advice. To the extent that it includes references to specific securities, commodities, currencies, or other instruments, those references do not constitute a recommendation to buy, sell or hold such security. This material does not and is not intended to take into account the particular financial conditions, investment objectives or requirements of individual customers. Before acting on this material, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice.
In accordance with EU regulation: The statements in this document shall not be considered as an objective or independent explanation of the matters. Please note that this document (a) has not been prepared in accordance with legal requirements designed to promote the independence of investment research, and (b) is not subject to any prohibition on dealing ahead of the dissemination or publication of investment research.
Any trading symbols displayed are for illustrative purposes only and are not intended to portray recommendations.
Disclosure: API Examples Discussed
Throughout the lesson, please keep in mind that the examples discussed are purely for technical demonstration purposes, and do not constitute trading advice. Also, it is important to remember that placing trades in a paper account is recommended before any live trading.
Disclosure: Order Types / TWS
The order types available through Interactive Brokers LLC’s Trader Workstation are designed to help you limit your loss and/or lock in a profit. Market conditions and other factors may affect execution. In general, orders guarantee a fill or guarantee a price, but not both. In extreme market conditions, an order may either be executed at a different price than anticipated or may not be filled in the marketplace.
Disclosure: Options Trading
Options involve risk and are not suitable for all investors. For more information read the Characteristics and Risks of Standardized Options, also known as the options disclosure document (ODD). To receive a copy of the ODD call 312-542-6901 or copy and paste this link into your browser:
Disclosure: Futures Trading
Futures are not suitable for all investors. The amount you may lose may be greater than your initial investment. Before trading futures, please read the CFTC Risk Disclosure. A copy and additional information are available at ibkr.com.