Golden Rule #6: Build Your Site for Humans and Robots

August 01, 2017 | By Tim Shiel

Download PDF


Your website has two different kind of visitors:

Humans, and Robots

A flaw of many web designs is that it only focuses on the primary users (humans) and thinks about the secondary users (robots) as, “out of sight, out of mind” and it’s hard to see (even with analytic tools) all those robots. Since building your site for humans is pretty obvious, this Golden Rule will focus on the not-so obvious users, build your site for robots. With no instructions, search engine robots (or, search engine spiders) start crawling every page it can sniff out from your domain. These spiders will most likely treat every page the same, often giving no preferential treatment to most important pages vs. least important pages. Unless, of course, you give these robots instructions.

Start with an XML Sitemap.

A human sitemap helps visitors see all the pages of a website. An XML sitemap does the same thing for the robots. XML sitemaps are easy to create by using sitemap generators. A quick Google search can point you in the right direction.

Restrict robot access to certain pages to increase SEO results with a Robots.txt file.

Think of robot as a new employee, if you think the new employee is going to find everything it needs without guidance to what is most important and what is least important, that employee is going to be very ineffective. Left to crawl on its own, the unguided robot will most likely focus as much on the troublesome areas of your website as it does the most important points. Restricting access to junk and hazards, while making quality choices easily accessible, is an important and often overlooked component of SEO. The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots, typically search engine robots, how to crawl and index pages on their website – and what NOT to crawl. Making this an easy but effective addition to any website.

Using a Canonical URL

Yes, it’s a bit of a strange word (Canonical, or as Google calls it, Canonicalization) but it’s worth understanding and making sure your site is properly coded for these robots. Canonicalization is simply the process of picking the best URL when there are several to choose from, and usually refers to home pages. For example, most people would consider these the same URLS:


So how do you make sure that Google picks the URL?


First, start by using that URL consistently across your entire site. For example, don’t make half of your links go to and the other half go to http://www.goldencomm. com. Instead, pick the URL you prefer and always use that format for your internal links. Next, configure your webserver so that if someone requests, it does a 301 (permanent) redirect to That helps robots know which URL you prefer to be canonical. Adding a 301 redirect can be an especially good idea if your site changes often (e.g. dynamic content, a blog, etc.).


Of course, make sure you’re using the proper meta-tags throughout the site.


For the purpose of this Golden Rule, we’ll focus on the three meta tags that are most commonly asked about. Title Tags – the mother of all meta tags and the easiest and most important single SEO tweak you can make. Title Tags have a real impact on search rankings and, perhaps just as importantly, are the only one of the tags we’ll discuss here that are visible to the average user. You’ll find them at the top of your browser (for organic search pages or for PPC landing pages):


Meta Description Attribute – can be a useful meta tag as it very simply explains to search engines and (sometimes) searchers what your page is about. Let’s say you were googling the phrase “SEO seminar” for example. You might encounter the following results:


The meta description tag is a snippet used to summarize a web page’s content – while it does NOT help in SEO results, it helps a ton for click-through rates as it lets the visitors know what a page is about before they click on it.

Alt Tags - (or Alt Attributes) are “alternative text” for an image. Alt tags are rarely seen by people (only SEO-type nerds) reading your website but the robots sure like them, and one way to “search” on Google is via images. So properly naming those images can be VERY important.


As more websites come online every day, and the world gets nosier and nosier, the robots need you to articulate clearly who your and your company are, and what you do. By providing structure to the chaos of the web, metadata helps robots (and people) find what they want and gives your content the edge in the world of search.



Based in Newport Beach in sunny Orange County, we have been serving Southern California and the Los Angeles area since our inception. Now serving clients nationwide. With four offices on three continents, we offer a global cadence to execute projects on time and on budget.

Orange County
World Headquarters
3420 Irvine Avenue
Newport Beach, CA 92660

(949) 570-6438 - sales
(949) 574-5500 - support
Salt Lake City
Development Office
634 South 400 West,
Suite 200
Salt Lake City, UT 84101

(801) 753-9974 - sales
Pune, India
.NET Development Office
4th Floor, Tower S4
Magarpatta City, Hadapsar
Pune - 411028
Wroclaw, Poland
PHP Development Office
Aleja Karkonoska 45 (2 piętro)
53-015 Wroclaw
This website uses cookies to ensure site visitors get the best experience on our website. By continuing to use this site, you accept our use of cookies and Privacy Statement. To learn more, please visit Privacy Statement