What is robotxt in SEO website?

Discuss Search Engine Optimization in relation to Joomla! 3.x. This forum will also have discussions on SEF/SEO Joomla! 3.x extensions.

Moderator: General Support Moderators

Forum rules
Forum Rules
Absolute Beginner's Guide to Joomla! <-- please read before posting, this means YOU.
Forum Post Assistant - If you are serious about wanting help, you will use this tool to help you post.
anayainterior
Joomla! Fledgling
Joomla! Fledgling
Posts: 1
Joined: Wed Jan 27, 2016 6:56 am

What is robotxt in SEO website?

Postby anayainterior » Wed Jan 27, 2016 7:01 am

Hi ,

I am anaya and i am SEO executive .
I want to know importance of roboext in seo website and how to create it ?
Please help me.

BY Anaya,
Last edited by pe7er on Wed Jan 27, 2016 4:10 pm, edited 1 time in total.
Reason: Manual signature has been removed

User avatar
AlexVega
Joomla! Hero
Joomla! Hero
Posts: 2093
Joined: Fri Aug 28, 2015 6:13 am
Location: México
Contact:

Re: What is robotxt in SEO website?

Postby AlexVega » Wed Jan 27, 2016 7:16 am

Hi there!

Welcome to the Forum, By default joomla! includes this file in the root
folder of your site.

Here you have more info:
https://docs.joomla.org/Robots.txt

Cheers!

minstudent
Joomla! Apprentice
Joomla! Apprentice
Posts: 29
Joined: Wed Sep 02, 2015 1:19 pm
Contact:

Re: What is robotxt in SEO website?

Postby minstudent » Wed Jan 27, 2016 10:59 pm

It's importen for you. You can search in Google: Generate robots.txt then you will have a lot of online tools to do it with.

BryanWalters
Joomla! Apprentice
Joomla! Apprentice
Posts: 32
Joined: Thu Dec 03, 2015 9:54 am

Re: What is robotxt in SEO website?

Postby BryanWalters » Thu Jan 28, 2016 3:52 pm

I would like to add the importance here.

This file will actually help you to control your sites data from crawling by search engine robots.
You can use tags created in automated tools.

User avatar
semnaveen
Joomla! Enthusiast
Joomla! Enthusiast
Posts: 104
Joined: Fri Jan 09, 2015 12:25 am
Location: Delhi
Contact:

Re: What is robotxt in SEO website?

Postby semnaveen » Thu Jan 28, 2016 8:38 pm

Hi Anaya,
Robots.txt works like a gatekeeper which control the visitors on the house. Same as it is robots.txt file works. It controls the spider to visit on your website pages. Here we we mention that on which pages should be crawled by the search engine crawlers and which not.
Especially, we block like login pages, admin pages, etc.

Thanks
Naveen

allielaine90
Joomla! Apprentice
Joomla! Apprentice
Posts: 23
Joined: Fri Feb 03, 2017 7:41 am
Contact:

Re: What is robotxt in SEO website?

Postby allielaine90 » Thu Feb 09, 2017 8:54 am

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

mathiasbund
Joomla! Apprentice
Joomla! Apprentice
Posts: 12
Joined: Tue Aug 11, 2015 3:00 pm

Re: What is robotxt in SEO website?

Postby mathiasbund » Thu Feb 09, 2017 9:18 am

The robots.txt file is important. It shows the Google crawlers what to look apart from and further what to index and make visible to tools like ahrefs etc.

gueenguarsmoza
Joomla! Intern
Joomla! Intern
Posts: 56
Joined: Mon Jun 13, 2016 1:17 am

Re: What is robotxt in SEO website?

Postby gueenguarsmoza » Sun Feb 12, 2017 1:12 am

As someone who doesn't have idea about it, I strongly suggest that you don't try to manipulate it. Well, it helps crawlers identify which part of your site should be accessed. If you want a page of your site to be uncrawled, then you can edit your robot.txt file.

JRmastering777
Joomla! Apprentice
Joomla! Apprentice
Posts: 32
Joined: Thu Feb 02, 2017 6:50 am
Location: Las Vegas, NV
Contact:

Re: What is robotxt in SEO website?

Postby JRmastering777 » Sun Feb 12, 2017 2:14 am

ROBOT.TXT file allows you to control what web crawlers (like Google) can't access on your website.

I say "can't" because if you don't have this file, the crawlers can access "everything." Nothing is being blocked. The web crawler can access all pages, directories, files, images, css, scripts, etc.

Joomla already comes with a robot.txt file that already blocks commonly blocked directories, but you can edited this to include more or less.

Why would you use this?
Some elements (folders) of your website you don't want made public on Google search. Like you logs, modules, plugins, your layouts, maybe your images, just to name a few.

How do you adjust items in the original Joomla robot.txt file?

Here are a "few" items included in the Joomla robot.txt file

User-agent: *

Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/

Web crawlers will no crawl the above directories. Let's say you want web crawlers to be able to crawl your cache directory (this is just an example). Just delete the item Disallow: /cache/

You can add directories. Disallow: /directory-name/

You can add individual pages and block them. Disallow: /page-name.html

You can also Wild card block characters. Disallow: /*block
In the above example, any page or directory (or file I believe) that contains the word "block" will not be crawled. Substitute the word "block" for whatever items you don't want crawled.

Note - Be careful with this. If you for example added Disallow: /*a , anything on the website with a letter A will not be crawled. That could be most of your website!! Ooops!
John Rogers :)

Online Mastering By JR Mastering - https://cdmusicmastering.com
Mixing & Mastering Video Courses - http://learnmixingmastering.com

Galleri
Joomla! Apprentice
Joomla! Apprentice
Posts: 24
Joined: Sat Jan 28, 2017 1:53 am
Contact:

Re: What is robotxt in SEO website?

Postby Galleri » Mon Feb 13, 2017 6:09 pm

It is crusial to have a robots.txt. Its the document that tells Google what to - and what not to index.

User avatar
lnuyasha
Joomla! Intern
Joomla! Intern
Posts: 87
Joined: Wed Mar 08, 2017 8:08 am
Contact:

Re: What is robotxt in SEO website?

Postby lnuyasha » Wed Mar 15, 2017 7:07 am

Robots.txt file is a text file containing the tail. txt that contains the commands to guide the robot to read google search and index file is selected when in website.
and is a simple text file (no HTML) are located in the root directory of the website
to create your robots.txt file, you open the program Notepad and type the command, then save it as robots.txt and copy it to the root directory of the website.

johnsimth774
Joomla! Apprentice
Joomla! Apprentice
Posts: 5
Joined: Thu Aug 03, 2017 6:41 am

Re: What is robotxt in SEO website?

Postby johnsimth774 » Fri Aug 11, 2017 11:56 am

The robot.txt file is a very powerful file that’s been created to instruct web robots how to crawl web pages on their website. It used by websites to communicate with web crawlers and other web robots.

tobias1975
Joomla! Fledgling
Joomla! Fledgling
Posts: 1
Joined: Sat Aug 12, 2017 8:21 am

Re: What is robotxt in SEO website?

Postby tobias1975 » Sat Aug 12, 2017 11:10 am

The robots.txt file should be placed in the root folder of your server. Like others already mentioned it is used to control web crawlers such as Google, Bing and everything else. Basically you can allow and disallow robots to crawl certain parts of your website. However Google nowadays needs to be able to crawl all your theme related files, css, js etc according to their new guidelines.

ashley_john
Joomla! Fledgling
Joomla! Fledgling
Posts: 1
Joined: Thu Aug 17, 2017 1:18 pm

Re: What is robotxt in SEO website?

Postby ashley_john » Thu Aug 17, 2017 1:33 pm

Web site owners use the "robots.txt" file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

deptraitunhien
Joomla! Apprentice
Joomla! Apprentice
Posts: 7
Joined: Sun Aug 20, 2017 9:23 am

Re: What is robotxt in SEO website?

Postby deptraitunhien » Sun Aug 20, 2017 9:38 am

Simply рut, robots.txt іѕ a соmmunісаtіоn mechanism bеtwееn the site аnԁ tһе search еngіnеѕ. It оutlіnеѕ the rulеѕ for іnԁехіng your website аnԁ рrеѕеntѕ them tо the web сrаwlеr. Ѕо basically, tһе file ԁеfіnеѕ which раrtѕ of уоur site Gооglе’ѕ allowed tо index аnԁ which ѕһоulԁ be аvоіԁеԁ.


Return to “Search Engine Optimization (Joomla! SEO) in Joomla! 3.x”

Who is online

Users browsing this forum: No registered users and 4 guests