How to get Tweets using Python and Twitter API v2 – Part III

QuantInsti

Contributor:
QuantInsti
Visit: QuantInsti

See Part I for an overview of the Tweepy client for Twitter API v2, and Part II to learn how to get Tweet(s) with Tweet Id(s) using client.

Pagination in client

We have mentioned earlier that the max_results parameter while searching tweets can have a maximum value of 100. So what do we do if we need more than 100 tweets? The answer is pagination.

Pagination in Client is similar to how we used Cursors for the API. Let us see an example of how we can fetch 1000 tweets. We have taken the same query that we used in the previous section.

Want another example? Here you go.

We have covered some of the important methods to fetch data from Twitter. There are many more to explore. If you would like to read about all the methods that are available, please read the official Client documentation ⁽⁷⁾.

The methods for the Client return a Response object with the results. In the subsequent sections, we will look at how we can save the search results to different data formats for analysis.


Using expansions to get user and media information

The Twitter API provides expansions ⁽⁸⁾ as a means to expand the payload we get from a search for a user or tweet lookup. Expansions help include additional data in the same response without the need for separate queries.

We can expand on the attachments.media_keys to view the media object, author_id to view the user object, and referenced_tweets.id to view the Tweet object the originally requested Tweet was referencing. The expanded objects are nested in the includes object.

Let us look at an example to get the user and media information using expansion:


Writing the search results to a text file

Let us now save the search results of a query to a text file.


Putting the search results into a DataFrame

Pandas DataFrames are great for working with a large amount of data. So let us save our search results as a Pandas DataFrame. If you would like to save this data in a csv file, you can use the Pandas to_csv() method.


Twitter API v2 GitHub

We have attempted to cover most of the common use cases for pulling Twitter data. If you want to explore the Twitter API v2 further, this Twitter Developer Platform Resources ⁽⁹⁾ repository is a great place to start. This repository has many sample codes ⁽¹⁰⁾ for the versatile functionality offered by the Twitter API v2.


Conclusion

In this two-part blog series, we explored how we can pull data from Twitter using the API and Client interfaces of the Tweepy library. We also looked at Twitter APIs and their functionalities.

Now that we have nicely organised data in Pandas DataFrames, we can perform natural language processing on it. Want to learn more? Don’t forget about our great course on Twitter Sentiment Analysis, check it out and go live trading!

Till then, happy coding!

Visit QuantInsti for additional insights on this topic: https://blog.quantinsti.com/twitter-api-v2/.

Disclosure: Interactive Brokers

Information posted on IBKR Traders’ Insight that is provided by third-parties and not by Interactive Brokers does NOT constitute a recommendation by Interactive Brokers that you should contract for the services of that third party. Third-party participants who contribute to IBKR Traders’ Insight are independent of Interactive Brokers and Interactive Brokers does not make any representations or warranties concerning the services offered, their past or future performance, or the accuracy of the information provided by the third party. Past performance is no guarantee of future results.

This material is from QuantInsti and is being posted with permission from QuantInsti. The views expressed in this material are solely those of the author and/or QuantInsti and IBKR is not endorsing or recommending any investment or trading discussed in the material. This material is not and should not be construed as an offer to sell or the solicitation of an offer to buy any security. To the extent that this material discusses general market activity, industry or sector trends or other broad based economic or political conditions, it should not be construed as research or investment advice. To the extent that it includes references to specific securities, commodities, currencies, or other instruments, those references do not constitute a recommendation to buy, sell or hold such security. This material does not and is not intended to take into account the particular financial conditions, investment objectives or requirements of individual customers. Before acting on this material, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice.

In accordance with EU regulation: The statements in this document shall not be considered as an objective or independent explanation of the matters. Please note that this document (a) has not been prepared in accordance with legal requirements designed to promote the independence of investment research, and (b) is not subject to any prohibition on dealing ahead of the dissemination or publication of investment research.

Any trading symbols displayed are for illustrative purposes only and are not intended to portray recommendations.

Disclosure: API Examples Discussed

Throughout the lesson, please keep in mind that the examples discussed are purely for technical demonstration purposes, and do not constitute trading advice. Also, it is important to remember that placing trades in a paper account is recommended before any live trading.