Get authenticated scraping working
This commit is contained in:
55
README.md
55
README.md
@ -1,15 +1,60 @@
|
||||
## Twitter scraper
|
||||
Scrape user's tweets :D
|
||||
|
||||
### Usage:
|
||||
## Usage:
|
||||
|
||||
`tweets = TweetsScraper().get_tweets_anonymous("<user_id>")`
|
||||
### Unauthenticated
|
||||
Example:
|
||||
```
|
||||
scraper = TweetScraper()
|
||||
tweets = scraper.get_tweets_anonymous("<user_id>")
|
||||
```
|
||||
|
||||
returns a list of tweets from the user as viewed from a logged-out session. Will only return 100 tweets (not necessarily the most recent)
|
||||
This will only allow use of the anonymous user tweets method, other methods will fail.
|
||||
|
||||
`tweets = TweetsScraper().get_tweets("<user_id>")`
|
||||
The anonymous method returns a list of tweets from the user as viewed from a logged-out session. It will only return 100 tweets (not necessarily the most recent)
|
||||
|
||||
not implemented yet, will get tweets as a logged in user
|
||||
|
||||
### Authenticated
|
||||
Example:
|
||||
```
|
||||
dotenv.load_dotenv()
|
||||
|
||||
auth_token = os.environ["AUTH_TOKEN"]
|
||||
csrf_token = os.environ["CSRF_TOKEN"]
|
||||
|
||||
scraper = TweetsScraper(auth_token, csrf_token)
|
||||
|
||||
user_id = scraper.get_id_from_screen_name("pobnellion")
|
||||
user_tweets = scraper.get_tweets(user_id, 100)
|
||||
```
|
||||
|
||||
Allows you to get tweets as a logged in user. Twitter only makes the 2000 ish most recent tweets available, but that should be more than enough.
|
||||
|
||||
You can either directly pass in the user id to `get_tweets()`, or use `get_id_from_screen_name()` to get the id if you don't have it.
|
||||
|
||||
To use dotenv, include a `.env` file in the directory with the following contents (no quotes around the values):
|
||||
```
|
||||
AUTH_TOKEN=<auth token>
|
||||
CSRF_TOKEN=<csrf token>
|
||||
```
|
||||
|
||||
You can find your auth and csrf tokens in twitter's cookies (F12 in your browser > storage tab > cookies)
|
||||
The auth token cookie is called `auth_token` and the csrf token is called `ct0`
|
||||
|
||||
### Tweet object
|
||||
Contains the text of the tweet, along with the timestamp and some stats (like count, repost count, views, etc)
|
||||
|
||||
#### Fields:
|
||||
- id : tweet id
|
||||
- views : view count
|
||||
- text : tweet content
|
||||
- likes : like count
|
||||
- replies : reply count
|
||||
- retweets : retweet count
|
||||
- quotes : quite tweet count
|
||||
- date : post date
|
||||
|
||||
Printing a tweet object results in an overview:
|
||||
|
||||
`L:52 RT:2 Q:1 R:3 V:1032 2025-01-20T01:53:57+00:00 Example tweet text`
|
||||
|
||||
Reference in New Issue
Block a user