#disallow

WIST Quotations Has Moved!wist@my-place.social
2025-05-22

A quotation from Ben Franklin

Sin is not hurtful because it is forbidden but it is forbidden because it’s hurtful.

Benjamin Franklin (1706-1790) American statesman, scientist, philosopher, aphorist
Poor Richard (1739 ed.)

Sourcing, notes: wist.info/franklin-benjamin/76…

#quote #quotes #quotation #qotd #franklin #benfranklin #benjaminfranklin #damaging #dangerous #disallow #forbid #hurting #sin

Arnaud Mangasaryan (Ramenos)ramenos@framapiaf.org
2024-02-26

Si le #bot d' #Openai respecte bien la directive #Disallow du fichier #robots.txt des sites, rien ne dit que les autres bot d'AI alternatifs la respectent :

tech.slashdot.org/story/24/02/

#artificialintelligence #crawling #website #bots

2024-01-05

If you have established Websites and wish to isolate them from search engines such as Google and other crawlers, place a text file named robots.txt with the following wording in the root directory of your Web site:

User-agent: *Disallow: /

The top line targets all crawlers, and the bottom line targets all files under the root directory for non-permission.

It is convenient because only two lines are needed to reject all crawlers, but placing this in place would leave all crawlers, which may harm search indexing and other aspects of the site. If there are any adverse effects, you can narrow the list of crawlers to reject and define them individually.

WordPress robots.txt can be edited safely using the following plug-ins without placing the text file directly in the root directory:

By the way; there is something I have been thinking about for a while that I would like to realize so that I will write it here.

Speaking of Cloudflare Tunnel, a revolutionary mechanism that allows you to securely expose your servers from inside your router rather than from the DMZ, and Cloudflare One/Cloudflare WARP, a combination of a gateway to securely access those contents on the go and a WireGuard client with MDM capabilities to connect to it.

For a long time, I have wanted to use Cloudflare Tunnel and Cloudflare WARP on the same server simultaneously, but that has been challenging to achieve. Sometimes, it degrades the Tunnel, or the server is unusable even when the Tunnel port is open.

I did a Google search and found the following exchange on a thread in the Cloudflare Community:

I have β€˜degraded status’ when using WARP with Zero Trust

β€œJust open the specified port,” the Cloudflare staff responded, but that didn’t work for me.

Now, I asked ChatGPT about it.

Using Cloudflare Tunnel and Cloudflare WARP on the same server can indeed be a bit challenging due to potential conflicts in network configurations and routing. However, it’s not impossible to set them up together. Here are some tips and considerations to help you achieve this:

  1. Port Conflicts:
    • Make sure that the ports used by Cloudflare Tunnel and Cloudflare WARP do not conflict. Check the default ports used by each service and configure them accordingly. You might need to adjust the settings for one of the services to use a different set of ports.
  2. Routing Issues:
    • Ensure that there are no conflicting routing rules between the two services. This can be particularly tricky because both services might try to manipulate the server’s network stack. Carefully review the routing tables to avoid conflicts.
  3. Firewall Settings:
    • Check your server’s firewall settings to ensure that it allows traffic for both Cloudflare Tunnel and Cloudflare WARP. If you are using iptables or another firewall tool, make sure that the necessary ports are open and traffic is allowed.
  4. Concurrency and Resources:
    • Consider the resource requirements of both services. Running both services simultaneously may consume a significant amount of system resources (CPU, memory, etc.). Ensure that your server has enough resources to handle the load. You can monitor resource usage using tools like top or htop.
  5. Logging and Debugging:
    • Enable logging for both Cloudflare Tunnel and Cloudflare WARP. Examine the logs for any error messages or warnings that might indicate conflicts or issues. This can help you pinpoint the source of the problem.
  6. Configuration Order:
    • Try starting the services in a specific order. For example, start one service first and ensure it’s working correctly before starting the second one. The order in which services are initialized can sometimes affect their behavior.
  7. Documentation and Support:
    • Refer to the official documentation of both Cloudflare Tunnel and Cloudflare WARP. Check if there are any specific guidelines or recommendations for running them together. Additionally, check the support forums or communities for both services to see if others have encountered similar issues.
  8. Separate Instances:
    • As a last resort, you might consider running Cloudflare Tunnel and Cloudflare WARP on separate instances or virtual machines. This ensures complete isolation and avoids potential conflicts.

Remember to make backups or snapshots of your server before making significant changes, and proceed with caution, especially if the server is in a production environment. If the issues persist, reaching out to Cloudflare support may provide additional assistance tailored to your specific setup.

If you ask a professional engineer, you will get this answer. But it’s the answer for someone who has never touched Cloudflare.

Does anyone know how to countermeasure against this β€œdegraded?”

https://kotaromiyasaka.com/search-engine-rejection-by-robots-txt-and-cloudflare-tunnel-failure/

#chatgpt #cloudflare #crawlers #degraded #disallow #firefish #google #mdm #one #plugin #robotsTxt #server #tunnel #warp #wireguard #wordpress #wprobotstxt #zerotrust

2023-08-10
ΒΏ Search .al a
robots.txt 
User-agent: *
Disallow:
User-agent: GPTBot
Disallow: /
2021-03-07

#letsroll

βš„ βš‚ βš€ βš… βš‚β†’#scale
⚁ βš‚ βš‚ βš„ βšƒβ†’#disallow
βš€ βš„ ⚁ βš‚ βš„β†’#cattail
βš€ βš„ ⚁ βšƒ βš‚β†’#catwalk
⚁ βš… βš‚ ⚁ βš…β†’#extradite
βš… βš… βš€ βš… βš‚β†’#washbasin

scale-disallow-cattail-catwalk-extradite-washbasin

Roll your own @ eff.org/deeplinks/2016/07/new-

2021-02-25

#letsroll

⚁ βš‚ βš‚ βš„ βšƒβ†’#disallow
βš… βš‚ βš„ βš„ βš„β†’#undying
βšƒ ⚁ ⚁ βš‚ βšβ†’#nucleus
βš… βš‚ βšƒ βš€ βš€β†’#uncoated
βš„ βš‚ βš… βšƒ βšβ†’#shale
βš€ βš„ ⚁ βš‚ βš€β†’#catlike

disallow-undying-nucleus-uncoated-shale-catlike

Roll your own @ eff.org/deeplinks/2016/07/new-

2021-01-15

#letsroll

βš‚ βš‚ βš… βš€ βšƒβ†’#headgear
βš… βš€ βš‚ ⚁ βš€β†’#support
βšƒ βš„ βš„ βš‚ βš…β†’#protozoan
βš‚ βšƒ βš… βš‚ βšβ†’#issuing
⚁ βš‚ βš‚ βš„ βšƒβ†’#disallow
⚁ βš„ βš€ βšƒ βšβ†’#embargo

headgear-support-protozoan-issuing-disallow-embargo

Roll your own @ eff.org/deeplinks/2016/07/new-

2021-01-15

#letsroll

βš… βš„ βš‚ βš‚ βš‚β†’#usher
⚁ βš‚ βš… ⚁ βš…β†’#dollhouse
βš€ ⚁ βš‚ βš‚ βš‚β†’#armoire
βš„ βš€ βš„ βšƒ βš‚β†’#reset
⚁ βš‚ βš‚ βš„ βšƒβ†’#disallow
βš€ βš… βš‚ βšƒ βš…β†’#coma

usher-dollhouse-armoire-reset-disallow-coma

Roll your own @ eff.org/deeplinks/2016/07/new-

2020-10-05

#letsroll

βš€ βš€ βš‚ βš… βšƒβ†’#aim
⚁ βš‚ βš‚ βš„ βšƒβ†’#disallow
βšƒ βš€ βš… βš„ βš„β†’#national
⚁ βš‚ βšƒ βšƒ βš„β†’#dismiss
βš… βš‚ βš„ βš‚ βš…β†’#underwire
βš‚ βš€ βš‚ βš„ βš‚β†’#fossil

aim-disallow-national-dismiss-underwire-fossil

Roll your own @ eff.org/deeplinks/2016/07/new-

2020-07-30

#letsroll

⚁ βš„ βš€ ⚁ βš€β†’#elk
⚁ ⚁ βšƒ βš„ βšβ†’#deflation
βš‚ βšƒ βš‚ βš€ βš‚β†’#hydroxide
βš… βš€ βš‚ βš… βšƒβ†’#suspect
⚁ βš‚ βš‚ βš„ βšƒβ†’#disallow
⚁ βš€ βš‚ βšƒ βš…β†’#creature

elk-deflation-hydroxide-suspect-disallow-creature

Roll your own @ eff.org/deeplinks/2016/07/new-

2020-05-24

#letsroll

βš€ ⚁ βš‚ βšƒ βš„β†’#arousal
⚁ βš‚ ⚁ βšƒ βš€β†’#dicing
⚁ βš‚ βš‚ βš„ βšƒβ†’#disallow
βšƒ βš‚ βšƒ βš€ βš„β†’#padding
βšƒ βš„ βš„ βš‚ βš‚β†’#protegee
βš€ βš€ ⚁ βš‚ βšβ†’#activity

arousal-dicing-disallow-padding-protegee-activity

Roll your own @ eff.org/deeplinks/2016/07/new-

2019-10-30

#letsroll

βš‚ βš‚ βš… βšƒ βšβ†’#heap
βš‚ βšƒ βš‚ ⚁ βš…β†’#hypnotize
βš… βš€ βš… βšƒ βš…β†’#taps
βš„ βš€ βš€ βš‚ βšƒβ†’#refuse
⚁ βš‚ βš‚ βš„ βšƒβ†’#disallow
βšƒ βš‚ βš‚ βš„ βš‚β†’#oxidize

heap-hypnotize-taps-refuse-disallow-oxidize

Roll your own @ eff.org/deeplinks/2016/07/new-

2019-10-26

#letsroll

⚁ ⚁ βš… βš€ βšβ†’#denim
⚁ βšƒ βšƒ βš„ βš„β†’#earring
⚁ βš… βš€ βšƒ βšβ†’#exclude
βš… βš‚ ⚁ βš… βšƒβ†’#unaware
⚁ βš‚ βš‚ βš„ βšƒβ†’#disallow
βšƒ βš€ βš… βšƒ βšβ†’#nanometer

denim-earring-exclude-unaware-disallow-nanometer

Roll your own @ eff.org/deeplinks/2016/07/new-

2019-06-24

#letsroll

βš… βš… βš‚ βšƒ βš…β†’#wilt
βš‚ βš… βšƒ βš„ βš…β†’#majesty
⚁ βš… βš€ βš‚ βšƒβ†’#exchange
βšƒ βš„ βš… ⚁ βš‚β†’#public
⚁ βš‚ βš‚ βš„ βšƒβ†’#disallow
βš€ βš‚ βš… βš€ βš€β†’#bonelike

wilt-majesty-exchange-public-disallow-bonelike

Roll your own @ eff.org/deeplinks/2016/07/new-

2019-04-21

#letsroll

⚁ βš€ βš‚ βš€ βš„β†’#craving
βš… βš… βš€ βš… βš„β†’#washbowl
βš€ βš‚ βš… βš‚ βš€β†’#booth
βš‚ βš‚ βš‚ βš„ βš‚β†’#handprint
βš… βš€ βš‚ βš€ βš„β†’#supplier
⚁ βš‚ βš‚ βš„ βšƒβ†’#disallow

washbowl-handprint-supplier-disallow-craving-booth

Roll your own @ eff.org/deeplinks/2016/07/new-

2019-03-26

#letsroll

βš„ βš€ βš„ βš… βš€β†’#resilient
βš‚ βš… βš‚ βš‚ βšƒβ†’#lumpish
βš„ βš‚ βšƒ βš„ βšβ†’#seizing
βšƒ βš„ ⚁ ⚁ βš€β†’#prefix
⚁ βš‚ βš‚ βš„ βšƒβ†’#disallow
βš€ ⚁ βš… βš‚ βšβ†’#babied

prefix-seizing-disallow-lumpish-babied-resilient

Roll your own @ eff.org/deeplinks/2016/07/new-

2019-03-08

#letsroll

βš„ βš‚ βš‚ βš… βšƒβ†’#scrunch
βšƒ βš… βš„ βš€ βš„β†’#rebate
⚁ βšƒ βšƒ βšƒ βš…β†’#earmark
⚁ βš‚ βš‚ βš„ βšƒβ†’#disallow
βš€ βš‚ ⚁ βš€ βš‚β†’#balancing
βš„ βš€ βš€ βš€ βš€β†’#reforest

rebate-earmark-balancing-reforest-scrunch-disallow

Roll your own @ eff.org/deeplinks/2016/07/new-

2019-01-26

#letsroll

βš‚ βšƒ βš„ βšƒ βšβ†’#improve
βš‚ βšƒ βš… βšƒ βš„β†’#jackal
βš€ ⚁ βšƒ ⚁ βš€β†’#asleep
βš„ βš€ βš€ βš„ βšβ†’#register
βš„ βš„ βš‚ βš„ βš‚β†’#species
⚁ βš‚ βš‚ βš„ βšƒβ†’#disallow

asleep-improve-jackal-register-disallow-species

Roll your own @ eff.org/deeplinks/2016/07/new-

2019-01-23

#letsroll

βš€ βš€ βš€ βš… βšβ†’#accustom
βšƒ βš‚ βš‚ βš… βšƒβ†’#pacemaker
⚁ βš‚ βš‚ βš„ βšƒβ†’#disallow
βš„ βš‚ βš„ ⚁ βšƒβ†’#sensation
⚁ ⚁ βš„ βš„ βšβ†’#demanding
βšƒ βšƒ βš… βšƒ βšƒβ†’#porridge

pacemaker-accustom-sensation-demanding-porridge-disallow

Roll your own @ eff.org/deeplinks/2016/07/new-

Client Info

Server: https://mastodon.social
Version: 2025.07
Repository: https://github.com/cyevgeniy/lmst