Link previews should respect robots.txt · Issue #21738 - GitHub
Server authors do not write robots.txt files with social link previews in mind, because no other social media software has used robots.txt to ...
Reduce load of preview fetching on third-party servers · Issue #23662
Fetch robots.txt (and cache it) for that remote URL. Link previews should respect robots.txt #21738; Lazily fetch the preview only when it's ...
TV Series on DVD
Old Hard to Find TV Series on DVD
Link preview should respect robots.txt · Issue #3242 - GitHub
Description When we send a link in Matrix, the client can use the Synapse's integrated prefetcher to fetch the link preview.
Store images and toots on IPFS to reduce inter-instance traffic and ...
Store images and toots on IPFS to reduce inter-instance traffic and local caching requirements #21461.
Astrid Sawatzky: "@shanselman @gme @crschmidt @t…" - Mastodon
GitHubLink previews should respect robots.txt · Issue #21738 · mastodon/mastodonSteps to reproduce the problem Share https://www.jefftk.com/test/no-robots or ...
Christopher Schmidt: "Fun fact: sharing this link on…" - Better Boston
First, robots.txt is a custom. It's not a rule. It's not a law. Second, plenty of bots and crawlers don't respect robots.txt. Some malicious, ...
Respect robots.txt / robots meta tag when crawling websites #1058
I'd suggest to implement some functionality to make the web crawler respect index / disallow settings as defined in the robots.txt or robots ...
Posit Connect Release Notes - Bookdown
Connect will serve a default robots.txt that disallows all crawling. Admins can specify a custom robots.txt using the Branding.Robots configuration option.
robots.txt blocks Twitter card preview #2774 - GitHub
Saved shot preview image and details are no longer displayed, only the saved shot link. [Notes]:. Here is a screenshot with the issue:
gitea/CHANGELOG.md at main · go-gitea/gitea - GitHub
Fixed robots.txt 404 error (#2226); Fix counts on issues dashboard (#2215); Fix unclosed session bug (#2214); Add collaborative repositories to the dashboard ...