RSS feeds getting caught in CloudFlare again

[kabi-chan]

My feed reader is getting stopped by CloudFlare's checks on nearly every request to either a story feed or to a list feed.

 

[2021]

Huh? How is that happening

 

[kabi-chan]

CloudFlare is, correctly, identifying that the feed reader isn't human and I guess the https://scribblehub.com/rssfeed.php URLs aren't whitelisted. So it's sending a challenge page instead of the actual RSS feed.

 

[kabi-chan]

This is still happening after several months and it's beyond annoying at this point. Is there anything that can be done to allow my feed reader to actually access the RSS feeds?

 

[Zirrboy]

Get a better reader?
I get past cloudflare with barebones code hacked together by yours truly in a few minutes (That does not make any attempt to hide its automated nature I am aware of). I haven't tested it over longer times of course, but not even spamming requests for a few minutes seemed to change any of that.

It's a hassle to migrate, but probably the quickest option available to you.

 

[kabi-chan]

Get a better reader?

It's an RSS reader. Getting a new one isn't going to change much of anything. It's a custom written reader that I made specifically with stories in mind anyway, so that's not really an option.

Up until I started having issues back in February, the reader always made itself known in the user agent (including a link to a page detailing what it was and the user agents and IP addresses it would use).

At this point, I can only assume that CloudFlare is just doing a straight IP block as no other client or script on my server is able to access Scribble Hub's RSS feeds.

I can't even fathom why an RSS feed would need to be protected in the first place. They should certainly be cached by CloudFlare, but preventing robots from accessing them doesn't make sense. RSS feeds are supposed be consumed by robots.

 

[Zirrboy]

It's an RSS reader. Getting a new one isn't going to change much of anything. It's a custom written reader that I made specifically with stories in mind anyway, so that's not really an option.

Up until I started having issues back in February, the reader always made itself known in the user agent (including a link to a page detailing what it was and the user agents and IP addresses it would use).

At this point, I can only assume that CloudFlare is just doing a straight IP block as no other client or script on my server is able to access Scribble Hub's RSS feeds.

I can't even fathom why an RSS feed would need to be protected in the first place. They should certainly be cached by CloudFlare, but preventing robots from accessing them doesn't make sense. RSS feeds are supposed be consumed by robots.

A RSS reader is a specialized HTTP client, so header setup can make a difference.
I use the following:

GET /rssfeed.php?[...] HTTP/2
User-Agent: Zirrboy
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive

And as for the blocking: If it's not a request fault, probably as you stated above. Non-exemption of the URLs. And since the feeds have disappeared from the novel pages I am inclined to believe they are far from being a core concern to the site's developer