Prerequisites
Before we begin, let's make sure we have the necessary tools in place.
First, download and install Node.js from the official website, making sure to use the Long-Term Support (LTS) version. This will also automatically install Node Package Manager (NPM) which we will use to install further dependencies.
For this tutorial, we will be using Visual Studio Code as our Integrated Development Environment (IDE) but you can use any other IDE of your choice. Create a new folder for your project, open the terminal and run the following command to set up a new Node.js project:
npm init -yThis will create a package.json file in your project directory, which will store information about your project and its dependencies.
Next, we need to install TypeScript and the type definitions for Node.js. TypeScript offers optional static typing which helps prevent errors in the code. To do this, run in the terminal:
npm install typescript @types/node --save-devYou can verify the installation by running:
npx tsc --versionTypeScript uses a configuration file called tsconfig.json to store compiler options and other settings. To create this file in your project, run the following command:
npx tsc -initMake sure that the value for “outDir” is set to “dist”. This way we will separate the TypeScript files from the compiled ones. You can find more information about this file and its properties in the official TypeScript documentation.
Now, create an “src” directory in your project, and a new “index.ts” file. Here is where we will keep the scraping code. To execute TypeScript code you have to compile it first, so to make sure that we don’t forget this extra step, we can use a custom-defined command.
Head over to the “package. json” file, and edit the “scripts” section like this:
"scripts": {
"test": "npx tsc && node dist/index.js"
}This way, when you will execute the script, you just have to type “npm run test” in your terminal.
Finally, to scrape the data from the website we will be using Puppeteer, a headless browser library for Node.js that allows you to control a web browser and interact with websites programmatically. To install it, run this command in the terminal:
npm install puppeteerIt is highly recommended when you want to ensure the completeness of your data, as many websites today contain dynamic-generated content. If you’re curious, you can check out before continuing the Puppeteer documentation to fully see what it’s capable of.
Data location
Now that you have your environment set up, we can start looking at extracting the data. For this article, I chose to scrape the Twitter profile of Netflix: https://twitter.com/netflix.
We’re going to extract the following data:
- the profile name;
- the profile handle;
- the user bio;
- the user location;
- the user website;
- the user join date;
- the user following count;
- the user followers count;
- information about the user tweets - author name - author handle - publishing date - text content - media (videos or photos) - reply count - retweet count - like count - view count.
You can see all this information highlighted in the screenshot below:
By opening the Developer Tools on each of these elements you will be able to notice the CSS selectors that we will use to locate the HTML elements. If you’re fairly new to how CSS selectors work, feel free to reach out to this beginner guide.
Extracting the data
Before writing our script, let’s verify that the Puppeteer installation went alright:
import puppeteer from 'puppeteer';
async function scrapeTwitterData(twitter_url: string): Promise<void> {
// Launch Puppeteer
const browser = await puppeteer.launch({
headless: false,
args: ['--start-maximized'],
defaultViewport: null
})
// Create a new page
const page = await browser.newPage()
// Navigate to the target URL
await page.goto(twitter_url)
// Close the browser
await browser.close()
}
scrapeTwitterData("https://twitter.com/netflix")Here we open a browser window, create a new page, navigate to our target URL and then close the browser. For the sake of simplicity and visual debugging, I open the browser window maximized in non-headless mode.
Now, let’s take a look at the website’s structure and extract the previous list of data gradually:
From the first glance you may have noticed that the website’s structure is pretty complex. The class names are randomly generated and very few HTML elements are uniquely identified.
Luckly for us, as we navigate through the parent elements of the targeted data, we find the attribute “data-testid”. A quick search in the HTML document can confirm that this attribute uniquely identifies the element we target.
Therefore, to extract the profile name and handle, we will extract the “div” element that has the “data-testid” attribute set to “UserName”. The code will look like this:
// Extract the profile name and handle
const profileNameHandle = await page.evaluate(() => {
const nameHandle = document.querySelector('div[data-testid="UserName"]')
return nameHandle ? nameHandle.textContent : ""
})
const profileNameHandleComponents = profileNameHandle.split('@')
console.log("Profile name:", profileNameHandleComponents[0])
console.log("Profile handle:", '@' + profileNameHandleComponents[1])Because both profile name and profile handle have the same parent, the final result will appear concatenated. To fix this, we use the “split” method to separate the data.
We then apply the same logic to extract the profile’s bio. In this case, the value for the “data-testid” attribute is “UserDescription”:
// Extract the user bio
const profileBio = await page.evaluate(() => {
const location = document.querySelector('div[data-testid="UserDescription"]')
return location ? location.textContent : ""
})
console.log("User bio:", profileBio)The final result is described by the “textContent” property of the HTML element.
Moving on the the next section of the profile’s data, we find the location, the website and the join date under the same structure.
// Extract the user location
const profileLocation = await page.evaluate(() => {
const location = document.querySelector('span[data-testid="UserLocation"]')
return location ? location.textContent : ""
})
console.log("User location:", profileLocation)
// Extract the user website
const profileWebsite = await page.evaluate(() => {
const location = document.querySelector('a[data-testid="UserUrl"]')
return location ? location.textContent : ""
})
console.log("User website:", profileWebsite)
// Extract the join date
const profileJoinDate = await page.evaluate(() => {
const location = document.querySelector('span[data-testid="UserJoinDate"]')
return location ? location.textContent : ""
})
console.log("User join date:", profileJoinDate)To get the number of following and followers, we need a slightly different approach. Check out the screenshot below:
There is no “data-testid” attribute and the class names are still randomly generated. A solution would be to target the anchor elements, as they provide an unique “href” attribute.
// Extract the following count
const profileFollowing = await page.evaluate(() => {
const location = document.querySelector('a[href$="/following"]')
return location ? location.textContent : ""
})
console.log("User following:", profileFollowing)
// Extract the followers count
const profileFollowers = await page.evaluate(() => {
const location = document.querySelector('a[href$="/followers"]')
return location ? location.textContent : ""
})
console.log("User followers:", profileFollowers)To make the code available for any Twitter profile, we defined the CSS selector to target the anchor elements whose “href” attribute ends with “/following” or “/followers” respectively.
Moving on to the list of tweets, we can again easily identify each one of them using the “data-testid” attribute, as highlighted below:
The code is no different than what we did until this point, with the exception of using the “querySelectorAll” method and converting the result to a Javascript array:
// Extract the user tweets
const userTweets = await page.evaluate(() => {
const tweets = document.querySelectorAll('article[data-testid="tweet"]')
const tweetsArray = Array.from(tweets)
return tweetsArray
})
console.log("User tweets:", userTweets)However, even though the CSS selector is surely correct, you may have noticed that the resulting list is almost always empty. That’s because the tweets are loaded a few seconds after the page has been loaded.
The simple solution to this problem is to add an extra waiting time after we navigate to the target URL. One option is to play around with a fixed amount of seconds, while another is to wait until a specific CSS selector appears in the DOM:
await page.waitForSelector('div[aria-label^="Timeline: "]')So, here we instruct our script to wait until a “div” element whose “aria-label” attribute starts with “Timeline: “ is visible on the page. And now the previous snippet should work flawlessly.
Moving on, we can identify the data about the tweet’s author just like before, using the “data-testid” attribute.
In the algorithm, we will iterate through the list of HTML elements and apply the “querySelector” method on each one of them. This way, we can better ensure that the selectors that we use are unique, as the targeted scope is much smaller.
// Extract the user tweets
const userTweets = await page.evaluate(() => {
const tweets = document.querySelectorAll('article[data-testid="tweet"]')
const tweetsArray = Array.from(tweets)
return tweetsArray.map(t => {
const authorData = t.querySelector('div[data-testid="User-Names"]')
const authorDataText = authorData ? authorData.textContent : ""
const authorComponents = authorDataText.split('@')
const authorComponents2 = authorComponents[1].split('·')
return {
authorName: authorComponents[0],
authorHandle: '@' + authorComponents2[0],
date: authorComponents2[1],
}
})
})
console.log("User tweets:", userTweets)The data about author will appear concatenated here as well, so to make sure that the result makes sense, we apply the “split” method on each section.
The text content of the tweet is pretty straightforward:
const tweetText = t.querySelector('div[data-testid="tweetText"]')
For the tweet’s photos, we will extract a list of “img” elements, whose parents are “div” elements with the “data-testid” attribute set to “tweetPhoto”. The final result will be the “src” attribute of these elements.
const tweetPhotos = t.querySelectorAll('div[data-testid="tweetPhoto"] > img')
const tweetPhotosArray = Array.from(tweetPhotos)
const photos = tweetPhotosArray.map(p => p.getAttribute('src'))
And finally, the statistics section of the tweet. The number of replies, retweets and likes are accessible in the same way, through the value of the “aria-label” attribute, after we identify the element with the “data-testid” attribute.
To get the number of views, we target the anchor element whose “aria-label” attribute ends with the “Views. View Tweet analytics” string.
const replies = t.querySelector('div[data-testid="reply"]')
const repliesText = replies ? replies.getAttribute("aria-label") : ''
const retweets = t.querySelector('div[data-testid="retweet"]')
const retweetsText = retweets ? retweets.getAttribute("aria-label") : ''
const likes = t.querySelector('div[data-testid="like"]')
const likesText = likes ? likes.getAttribute("aria-label") : ''
const views = t.querySelector('a[aria-label$="Views. View Tweet analytics"]')
const viewsText = views ? views.getAttribute("aria-label") : ''Because the final result will also contain characters, we use the “split” method to extract and return the numerical value only. The complete code snippet to extract tweets data is shown below:
// Extract the user tweets
const userTweets = await page.evaluate(() => {
const tweets = document.querySelectorAll('article[data-testid="tweet"]')
const tweetsArray = Array.from(tweets)
return tweetsArray.map(t => {
// Extract the tweet author, handle, and date
const authorData = t.querySelector('div[data-testid="User-Names"]')
const authorDataText = authorData ? authorData.textContent : ""
const authorComponents = authorDataText.split('@')
const authorComponents2 = authorComponents[1].split('·')
// Extract the tweet content
const tweetText = t.querySelector('div[data-testid="tweetText"]')
// Extract the tweet photos
const tweetPhotos = t.querySelectorAll('div[data-testid="tweetPhoto"] > img')
const tweetPhotosArray = Array.from(tweetPhotos)
const photos = tweetPhotosArray.map(p => p.getAttribute('src'))
// Extract the tweet reply count
const replies = t.querySelector('div[data-testid="reply"]')
const repliesText = replies ? replies.getAttribute("aria-label") : ''
// Extract the tweet retweet count
const retweets = t.querySelector('div[data-testid="retweet"]')
const retweetsText = retweets ? retweets.getAttribute("aria-label") : ''
// Extract the tweet like count
const likes = t.querySelector('div[data-testid="like"]')
const likesText = likes ? likes.getAttribute("aria-label") : ''
// Extract the tweet view count
const views = t.querySelector('a[aria-label$="Views. View Tweet analytics"]')
const viewsText = views ? views.getAttribute("aria-label") : ''
return {
authorName: authorComponents[0],
authorHandle: '@' + authorComponents2[0],
date: authorComponents2[1],
text: tweetText ? tweetText.textContent : '',
media: photos,
replies: repliesText.split(' ')[0],
retweets: retweetsText.split(' ')[0],
likes: likesText.split(' ')[0],
views: viewsText.split(' ')[0],
}
})
})
console.log("User tweets:", userTweets)After running the whole script, your terminal should show something like this:
Profile name: Netflix
Profile handle: @netflix
User bio:
User location: California, USA
User website: netflix.com/ChangePlan
User join date: Joined October 2008
User following: 2,222 Following
User followers: 21.3M Followers
User tweets: [
{
authorName: 'best of the haunting',
authorHandle: '@bestoffhaunting',
date: '16 Jan',
text: 'the haunting of hill house.',
media: [
'https://pbs.twimg.com/media/FmnGkCNWABoEsJE?format=jpg&name=360x360',
'https://pbs.twimg.com/media/FmnGkk0WABQdHKs?format=jpg&name=360x360',
'https://pbs.twimg.com/media/FmnGlTOWABAQBLb?format=jpg&name=360x360',
'https://pbs.twimg.com/media/FmnGlw6WABIKatX?format=jpg&name=360x360'
],
replies: '607',
retweets: '37398',
likes: '170993',
views: ''
},
{
authorName: 'Netflix',
authorHandle: '@netflix',
date: '9h',
text: 'The Glory Part 2 premieres March 10 -- FIRST LOOK:',
media: [
'https://pbs.twimg.com/media/FmuPlBYagAI6bMF?format=jpg&name=360x360',
'https://pbs.twimg.com/media/FmuPlBWaEAIfKCN?format=jpg&name=360x360',
'https://pbs.twimg.com/media/FmuPlBUagAETi2Z?format=jpg&name=360x360',
'https://pbs.twimg.com/media/FmuPlBZaEAIsJM6?format=jpg&name=360x360'
],
replies: '250',
retweets: '4440',
likes: '9405',
views: '656347'
},
{
authorName: 'Kurtwood Smith',
authorHandle: '@tahitismith',
date: '14h',
text: 'Two day countdown...more stills from the show to hold you over...#That90sShow on @netflix',
media: [
'https://pbs.twimg.com/media/FmtOZTGaEAAr2DF?format=jpg&name=360x360',
'https://pbs.twimg.com/media/FmtOZTFaUAI3QOR?format=jpg&name=360x360',
'https://pbs.twimg.com/media/FmtOZTGaAAEza6i?format=jpg&name=360x360',
'https://pbs.twimg.com/media/FmtOZTGaYAEo-Yu?format=jpg&name=360x360'
],
replies: '66',
retweets: '278',
likes: '3067',
views: ''
},
{
authorName: 'Netflix',
authorHandle: '@netflix',
date: '12h',
text: 'In 2013, Kai the Hatchet-Wielding Hitchhiker became an internet sensation -- but that viral fame put his questionable past squarely on the radar of authorities. \n' +
'\n' +
'The Hatchet Wielding Hitchhiker is now on Netflix.',
media: [],
replies: '169',
retweets: '119',
likes: '871',
views: '491570'
}
]Scaling up
While scraping Twitter may seem easy at first, the process can become more complex and challenging as you scale up your project. The website implements various techniques to detect and prevent automated traffic, so your scaled-up scraper starts getting rate-limited or even blocked.
One way to overcome these challenges and continue scraping at large scale is to use a scraping API. These kinds of services provide a simple and reliable way to access data from websites like twitter.com, without the need to build and maintain your own scraper.
WebScrapingAPI is an example of such a product. Its proxy rotation mechanism avoids blocks altogether, and its extended knowledge base makes it possible to randomize the browser data so it will look like a real user.
The setup is quick and easy. All you need to do is register an account, so you’ll receive your API key. It can be accessed from your dashboard, and it’s used to authenticate the requests you send.
As you have already set up your Node.js environment, we can make use of the corresponding SDK. Run the following command to add it to your project dependencies:
npm install webscrapingapiNow all it’s left to do is to send a GET request so we receive the website’s HTML document. Note that this is not the only way you can access the API.
import webScrapingApiClient from 'webscrapingapi';
const client = new webScrapingApiClient("YOUR_API_KEY");
async function exampleUsage() {
const api_params = {
'render_js': 1,
'proxy_type': 'residential',
'wait_for_css': 'div[aria-label^="Timeline: "]',
'timeout': 30000
}
const URL = "https://twitter.com/netflix"
const response = await client.get(URL, api_params)
if (response.success) {
console.log(response.response.data)
} else {
console.log(response.error.response.data)
}
}
exampleUsage();By enabling the “render_js” parameter, we send the request using a headless browser, just like you previously did along with this tutorial.
After receiving the HTML document, you can use another library to extract the data of interest, like Cheerio. Never heard of it? Check out this guide to help you get started!
Conclusion
This article has presented a comprehensive guide on how to web scrape Twitter effectively using TypeScript. We covered the steps of setting up the necessary environment, locating and extracting data, and the potential uses of this information.
Twitter is a valuable source of data for those looking to gain insights into consumer sentiment, social media monitoring, and business intelligence. However, it's important to note that using the Twitter API alone may not be enough to access all the data you need, and that's why using a professional scraper is a better solution.
Overall, web scraping Twitter can provide valuable insights and can be a great asset for any business or individual looking to gain a competitive edge.
