/r/pushshift

Photograph via snooOG

Subreddit for users of the pushshift.io API

Rules:

  • Please be kind to each other.
  • Please see this thread before posting "Is Pushshift Down?"
  • Camas, RedditSearch, and similar tools are unlikely to return due to recent API made by Reddit changes. Please do not post asking about those tools.

Want your data removed, use this Removal Request Form

/r/pushshift

14,389 Subscribers

1

Method Not Allowed error

I've been getting this error for the past couple days. I had access in the past. Is there anything I can do to fix the issue? Or is it happening to others.

This is after trying to authorize from https://api.pushshift.io/signup

0 Comments
2024/09/08
22:28 UTC

0

Any clue why I get this when I try to authenticate?

{"detail":"User is not an authorized moderator."}

{"detail":"User is not an authorized moderator."}

3 Comments
2024/09/04
22:31 UTC

3

Need Access for Research

Hi all,

I want to access the reddit data using pushshift API. I raised a request. Can anyone help me how can I get the access at the earliest?

Thanks1

11 Comments
2024/09/04
16:58 UTC

1

Gab data for research purpose.

Hi, I've been searching for a dataset containing Gab posts. I finally came across a link but there is a login page coming up. I signed up and logged in, but since there is another guardrail requiring approval of requests and requests can only be submitted by moderators. I am unable to get access.

Is there any way of getting access to the data through my researcher credentials.

8 Comments
2024/08/25
03:51 UTC

2

Help with handling big data sets

Hi everyone :) I'm new to using big data dumps. I downloaded the r/Incels and r/MensRights data sets from u/Watchful1 and are now stuck with these big data sets. I need them for my Master Thesis including NLP. I just want to sample about 3k random posts from each Subreddit, but have absolutely no idea how to do it on data sets this big and still unzipped as a zst (which is too big to access). Has anyone a script or any ideas? I'm kinda lost

6 Comments
2024/08/22
09:33 UTC

1

How can I view a deleted post

I'm not a programmer, but I know that Pushshift functions as an archive for Reddit. Many posts I've interacted with have been deleted, and sometimes I'd like to see what the original post said. How can I view it?

Additionally, sometimes the post itself isn't deleted, but the original poster's account is gone, and I want to remember who made the post.

1 Comment
2024/08/06
17:13 UTC

0

Action Needed: Reauthorization of API access

Hello all,

Earlier this week, Pushshift faced a breach of security because of which the application configuration had to be updated. The updated application that authorizes you now goes by the name "ncri_ingest". All users will need to reauthorize for API access through https://api.pushshift.io/signup.

Users that have a long-running script using the refresh functionality will also need to replace the token with a new one after reauthorizing.

We apologize for any inconvenience caused and appreciate your patience during this period.

  • On behalf of Team NCRI
11 Comments
2024/08/01
00:00 UTC

9

FYI: Reddit is scaling up their "Reddit for Researchers" program

1 Comment
2024/07/31
21:46 UTC

8

Error code when trying to reauthorize

When it goes to the reddit page, I get;

bad request (reddit.com)

you sent an invalid request

— invalid client id.

11 Comments
2024/07/30
16:42 UTC

2

How long does it take Pushshift to respond to removal requests?

Requested nearly a week ago, I’ve heard nothing.

4 Comments
2024/07/18
13:44 UTC

7

Does pushshift support need to be notified when it's down?

I've just starting using it again recently - what's the protocol? Does it go down often?

It's been down for me for a few days now.

3 Comments
2024/07/14
15:32 UTC

27

Reddit dump files through July 2024

https://academictorrents.com/details/20520c420c6c846f555523babc8c059e9daa8fc5

I've uploaded a new centralized torrent for all monthly dump files through the end of July 2024. This will replace my previous torrents.

If you previously seeded the other torrents, loading up this torrent should recheck all the files (took me about 6 hours) and then download only the new files. Please don't delete and redownload your old files.

18 Comments
2024/07/13
03:55 UTC

2

Indexing Pushshift

Hi all,

I am a researcher and I used to collect Pushshift data using the API. Now I need to collect data again. The issue is I do not need a specific subreddit bu specific posts that cotain targeted expression and then I need to collect posts of that user who made these comments. Let's say in the last 5 years.
I was thinking to index the data in our lap (the last 5-6 years of pushshift comments and posts)
Did any one do that before or is there any guide or project for this so it saves the time experimenting with tools and structure?

Edit: What I mean exactly is if you have indexd Pushshift data youself what did you use, MongoDB / Elasticsearch?
Any one have docker file / code that get me started with this task faster?

Thanks,

Kind regards

7 Comments
2024/07/11
11:42 UTC

2

Confirmation of an account being removed?

Anyone know how we can get confirmation an account was removed after we submit the request? I can see the link to submit it but I don't see how we would get notified once it happened? Or maybe someone knows what website I could check?

2 Comments
2024/06/22
17:42 UTC

2

Not all PushShift shards are active

I'm trying to use the PushshiftAPI() and it gives the following error: WARNING:pmaw.PushshiftAPIBase:Not all PushShift shards are active. Query results may be incomplete.

why it's not working? what can I do?

0 Comments
2024/06/13
16:50 UTC

6

system stuck in an authentication loop

i accept the terms, i allow access, i get the search interface

but then when i try to search i get a pop up saying authentication is required and i am back to square one.

3 Comments
2024/06/03
16:42 UTC

1

Help with Finding A Guide

So first off id like to say appreciate you guys doing this. It's thankless work and really cool for people looking for long gone stuff so thank you 🙏

Now on to my problem . I won't rule out that what I'm about to ask is easy and I'm just not familiar enough with json files to know , so if it is , please be easy on my as I have tried frrsearching on my on and their post is a last ditch effort.

So there is a guide / tutorial that was posted a while back in an now deleted sub reddit. I have downloaded both the " posts " and " comments " dumps and tried searching through them using notepad++ and the search function. I have found numerous instances of the name of the guide , but have yet to find the full guide post itself.

Is there an easier way to try and find it? When I do get a hit , they all look to be 1 line long and that's it. Any tips trick or anything I need to do different to find the full guide I'm looking for?

Thanks in advance to anyone that can off anything. It's greatly appreciated 🙏

2 Comments
2024/05/29
01:14 UTC

3

SERVICE RESTORED: Recent data issues with Pushshift

Hello all,

We observed downtimes in Pushshift and occasional failure to collect data for the last few days. On diagnosis, this was owing to an internal server and storage issue. The system was fixed this morning, and data is now being collected normally. We appreciate your patience and apologize for any inconvenience caused during this period.

-Pratik

On behalf of Team Pushshift

6 Comments
2024/05/24
21:17 UTC

1

Pushshift is currently broke for mobile using chrome in desktop mode.

It looks like I can no longer grab the access cookie to allow access on mobile with chrome in desktop mode (android os).

It looks to be two issues:

  • The "Sign in with Reddit" button does not allow a long press to open as a tab and therefore allow the cookie to go into my chrome app.

  • Clicking the button opens the Reddit App and the built in browser. A recent update looks to have removed their option to "open in chrome" from that built in browser. This means I can no longer use that button to force the access page to go back into the chrome app.

Please can the devs either fix the button to allow opening in a tab on the chrome mobile app, or ask Reddit to add back in the "open in chrome" button for the official Reddit apps in-built website browser?

1 Comment
2024/05/24
08:16 UTC

11

Dump files for April 2024

April dump files: https://academictorrents.com/details/9b29491dccf7d9d72e5538ce8b647cf8ed43fb34

Sorry for the delay a second month in a row, still working on my upload process.

2 Comments
2024/05/24
03:10 UTC

4

Ingest seems to have stalled ~36 hours ago

Hello,

PushShift ingest seems to have stalled around
Mon May 20 2024 21:49:29 GMT+0200

The frontend is up & responding with hits older than that.

Is this just normal maintenance?

Regards

2 Comments
2024/05/22
06:40 UTC

1

Does anyone have a script that maps posts to comments >

Long shot but does anyone have a script out there that maps posts to comments, and combines them in a new json object. from the dumps I've collected like 25k posts and 75k comments and since they are kinda random rn, I would like to map posts to comments to do some better analysis

13 Comments
2024/05/19
20:33 UTC

1

"User is not an authorized moderator."

I keep getting this message despite 1) being a moderator and 2) having received approval from pushshift.

does anyone know how to resolve this?

8 Comments
2024/05/14
19:12 UTC

0

Emergency

Postgrad student who's (academic) life is hanging on a thread if she failed to use PRAW or Pushift to scrape comments from subreddit 'r/gameofthrones'!!!!!!!!

https://preview.redd.it/5px05uzt620d1.png?width=1654&format=png&auto=webp&s=3fe52df48862188c4bd37b3d7bba54985be63583

15 Comments
2024/05/12
20:54 UTC

5

Trouble with zst to csv

Been using u/watchful1's dumpfile scripts in Colab with success, but can't seem to get the zst to csv script to work. Been trying to figure it out on my own for days (no cs/dev/coding background), trying different things (listed below), but no luck. Hoping someone can help. Thanks in advance.

Getting the Error:

IndexError                                Traceback (most recent call last)


 in <cell line: 50>()
     52                 input_file_path = sys.argv[1]
     53                 output_file_path = sys.argv[2]
---> 54                 fields = sys.argv[3].split(",")
     55 
     56         is_submission = "submission" in input_file_path

<ipython-input-22-f24a8b5ea920>

IndexError: list index out of range

From what I was able to find, this means I'm not providing enough arguments.

The arguments I provided were:

input_file_path = "/content/drive/MyDrive/output/atb_comments_agerelat_2123.zst"
output_file_path = "/content/drive/MyDrive/output/atb_comments_agerelat_2123"
fields = []

Got the error above, so I tried the following...

  1. Listed specific fields (got same error)

input_file_path = "/content/drive/MyDrive/output/atb_comments_agerelat_2123.zst"
output_file_path = "/content/drive/MyDrive/output/atb_comments_agerelat_2123"
fields = ["author", "title", "score", "created", "id", "permalink"]

  1. Retyped lines 50-54 to ensure correct spacing & indentation, then tried running it with and without specific fields listed (got same error)

  2. Reduced the number of arguments since it was telling me I didn't provide enough (got same error)

    if name == "main": if len(sys.argv) >= 2: input_file_path = sys.argv[1] output_file_path = sys.argv[2] fields = sys.argv[3].split(",")

    No idea what the issue is. Appreciate any help you might have - thanks!

18 Comments
2024/05/11
22:59 UTC

Back To Top