|
|
|
Internet News Feeds |
|
SEOmoz Daily SEO Blog
|
SEOmoz, a Seattle-based search engine optimization company, serves as a hub for search marketers worldwide, providing education, tools, resources and paid services.
|
-
Roundup Thursday for the Week of 6/29/08
Posted by rebecca Stories, news, and other notable items from the past week:
One star link:
- Hacker News started a thread asking why hackers hate SEO. Most of the answers are frustrating, though some folks chime in to defend our industry.
Three star links:
- The fellas at Nordic eMarketing went duck herding in SEOmoz t-shirts. Glad to see our shirts were put to good use!
- Aaron Wall thinks that the best way for Microsoft to compete with Google is to buy some big brands/sites that dominate Google's organic rankings, such as Expedia, Monster.com, WebMD, the NY Times, etc.
- Online purchases are becoming more diverse, according to an eMarketer survey. However, buyers still have privacy and security concerns, which is unsurprising.
- Search Engine Roundtable reports that Google thinks blocking countries outside of the US is considered cloaking and would be against Google's guidelines. However, they've since retracted their stance and now say that it would be geolocation. Stop messing with our heads, Google!
- Speaking of cloaking, Danny Sullivan weighs in on the "white hat cloaking" debate.
- Onward Search put together a handy checklist that can help prepare you for an SEO interview. I like how they added "wash and dress" to the list--it's a pretty important step, all right!
Four star links:
Five star links:
- Over at SEO Scoop, James Duthie examines the traffic quality between SEO and SMO and concluded that "search engines generate superior quality traffic." It's important to keep in mind that while social media marketing is still a great tactic to leverage, it's not a replacement for natural search engine traffic.
YOUmoz entries:
Best of YOUmoz:
New events added to the Events Calendar:
Upcoming events:
New additions to the SEOmoz Marketplace:
Featured job postings:
Featured companies:
United States/North America:
UK/Europe:
Australia:
Featured resumes:
Currently looking:
- Austin Wisner is an SEO and PPC marketer who also has extensive writing experience.
- David Arrington is the founder and CEO of Blue Fusion Marketing and has expertise in running a successful online marketing company.
- Joe Williams is seeking an organic SEO position with an SEO/marketing firm or long-term contract work with an organization interested in moving to the top of major search engines. He has experience with building SEO-optimized websites and integrating e-commerce packages, as well as SEO knowledge.
- Babul Paul is an SEO/SEM manager who is looking to leverage his experience of Internet marketing, web development, and email marketing.
Happily employed:
- Samual is an SEM expert with over two years' experience in Internet marketing and SEO.
Do you like this post? Yes No
-
Reddit, Stumbleupon, Del.icio.us and Hacker News Algorithms Exposed!
Posted by Danny Dover It is greatly ironic that algorithms, the quintessential example of all that is not human, would be so fundamental to social media. Last week I wrote a post about how Google gathers user data. This week I continue by exposing how popular social media websites use algorithms to utilize user data.
Although humans power social media, it is algorithms that provide the frameworks that make user input useful. As proven by the countless social sites online, finding the correct mix of participation and rules can be extremely difficult. Below are some of the algorithms that when combined with the right people have proven successful.
Popular Social Media Algorithms
Y Combinator's Hacker News:
Formula:
(p - 1) / (t + 2)^1.5
Description:
Votes divided by age factor
p = votes (points) from users.
t = time since submission in hours.
p is subtracted by 1 to negate submitters vote.
age factor is (time since submission in hours plus two) to the power of 1.5.
Source: Paul Graham, creator of Hacker News
Reddit:
Formula:
Description:
First of all, the time 7:46:43 am on December 8th 2005 is a constant used to determine the relative age of a submission. (It is likely the time the site launched but I have not been able to confirm this) The time the story was submitted minus the constant date is ts. ts works as the force that pulls the stories down the frontpage.
y represents the relationship of up votes to down votes.
45000 is the amount of seconds in 12.5 hours. This constant is used in combination with yts to "water down" votes as they are made farther and farther from the time the article was submitted.
log10 is also used to make early votes carry more weight than late votes. In this case, the first 10 votes have exactly as much weight as votes 11 through 101.
Source: code.reddit.com, Redflavor.com and Hacker News user Aneesh
StumbleUpon:
Formula:
(Initial stumbler audience / # domain) + ((% stumbler audience / # domain) + organic bonus – nonfriend) – (% stumbler audience + organic bonus) + N
Description:
The initial stumbler "power" (Audience of the initial stumbler divided by the amount of times that stumbler has stumbled the given domain) is added to the sum of all the subsequent stumbler's powers.
Subsequent stumbler power is ((Percentage of audience stumbler makes up divided by the number of times given stumbler has stumbled domain) + a predetermined power boost for using the toolbar - a predetermined power drain if stumblers are connected) + (% of the stumbler audience + a predetermined boost for using the toolbar)
N is a "safety variable" so that the assumed algorithm is flexible. It represents a random number.
Source: 2007 Tim Nash at The Venture Skills Blog Please see his blog post for more in depth information
Del.icio.us:
Formula:
Points = (Amount of times story has been bookmarked in the last 3600 seconds)
Description:
Rank on Del.icio.us Popular is determined by comparing points. Points represent the amount of times a story has been bookmarked in the last hour. The higher the rate, the higher the points. Every bookmark counts as one point.
3600 is the seconds in one hour.
Source: Based on my extended observations of Del.icio.us Popular
Avoiding the 10,000 lb Gorilla in the Room (Digg.com)
Digg is different. The company is a lot less transparent than the above mentioned companies. It is fearful of being gamed and in response has created a secritive algorithm that appears to be far more complex than its competition.
At a minimum I expect that Digg's algorithm takes into account the following factors:
- Submission Time
- Submission Category
- Submitter's Digg authority
- Submitter's website wide activity
- Sumbitter's friends and fans
- Subsequent digger's authority
- Subsequent digger's friends and fans
- Subsequent digger's geo location
- Subsequent digger's HTTP referer
I look forward to hearing from the social media experts in the comments. Please let me know how I can improve this post. If any of you (experienced or inexperienced) decide it would be preferable to contact me privately, please feel free to via E-mail, Linkedin or Twitter. Thanks!
Do you like this post? Yes No
-
Cracking Google's 1,000 Page Barrier
Posted by Dr. Pete One of the frustrations of doing SEO for large websites is the fact that Google makes it very difficult to see more than a small part of the search index. Even in Webmaster Tools, Google's index search is built on the same mechanics as its web search, which only lets you see the first 1,000 pages of any result. Whether you're trying to get pages discovered, struggling with duplicate content, confirming robots.txt changes, or doing advanced index sculpting, that 1,000-page barrier can be extremely limiting when you're dealing with a site with 10,000 or more indexed pages.
So, how can we dig deeper into the index and really see the big picture?
The Tools – Site: and Inurl:
First off, you're going to need a couple of tools. I'll assume that most of you are familiar with Google's "site:" command, which returns the indexed pages from any given domain or subdomain. Let's take our friends here at SEOmoz as an example. Type "site:seomoz.org" into Google's search box, and you'll see something like this:
The other command we'll be using is "inurl:", which, paired with other search terms, restricts the results to only those containing a specific keyword in the URL. Paired with the "site:" command, Google only reveals indexed pages which contain those URL keywords.
The Tactic – Index Deconstruction
Using our SEOmoz example, how can we find out which pages are included in the roughly 12,000-page index when we can only see those pages 1,000 at a time? Those last three words are the key: we can only see 1,000 pages at a time, but depending on how we construct our searches, they don't have to be the same 1,000 pages. By splitting up our index searches logically, we can break the full index up into manageable chunks. We'll do this by using "inurl:" to force the "site:" command to show us the index through smaller windows.
An Example – Deconstructing SEOmoz
This is one of those techniques that's much easier to illustrate with an example. Let's say that we needed to dig deeply into SEOmoz's 12,000 indexed pages. The first thing that we might do is to take a look at the main navigation to get an idea of the URL/folder structure of the site. Looking at the top-right navigation on SEOmoz, we see the following (I've added the numbers 1-6 - see below):
Other than "Home," the first link goes to the "/blog" folder. That looks promising, so let's try out our combination "site:" and "inurl:" search:
After clicking the "omitted results" link to see the full list, we get 2,430 pages of the index that contain the word "blog." That's a good start, so let's see what we can do with a few more of the major folders (numbered above):
- inurl:blog – 2430
- inurl:ugc - 712
- inurl:articles - 96
- inurl:tools - 29
- inurl:users – 5880
- inurl:marketplace - 787
Not bad: with just 6 subfolders, we've accounted for 9,934 pages or over 80% of the index. This, of course, assumes minimal overlap, and the accuracy of Google's numbers may be questionable (I'll discuss some issues with "inurl:" at the end of the post), but it's more than adequate to get the job done.
Now, we're left with a couple of groups, such as (5) that are still greater than 1,000 pages. At this point, you'll have to use some logic and your knowledge of the site in question. As a frequent Moz user, I know that the "users" folder contains all of the user profiles. Digging a little, I can easily find that those profiles all contain "users/view." A new search on "inurl:users/view" reveals 5,810 user profiles, making up almost all of the pages in the "users" folder and almost half of the total index.
An Example – Canonical URLs
Most of the time, we aren't going to be trying to deconstruct the entire Google index for a site, but just need to answer a specific question. Let's take my own company site/blog as an example. Recently, I realized that I had left some loose ends in the code that were revealing both canonical and non-canonical URLs. So, for example, the same blog post might have the following two URLs:
- http://www.usereffect.com/topic/the-last-spam-youll-ever-need
- http://www.usereffect.com/index.php?id=154
I've recently made some code changes to fix the problem, but how do I find out if my fix is working? I simply look for "id" in the URL with a search command like "site:usereffect.com inurl:id". As of this writing, that search only shows 1 result, suggesting that my changes are having the desired effect.
Advanced Inurl Tips
I hope that I've demonstrated just how powerful two relatively simple search tools can be when effectively combined. Before you go out and put this to work, though, a couple of warnings about "inurl:", which has a tendency to misbehave.
First, "inurl:" seems to ignore punctuation, for the most part. A targeted search on the folder "inurl:/blog" returns the same results as "inurl:blog," which is to say that it returns every page that contains "blog" anywhere in the URL. In some cases, this won't be a problem, but you'll have to judge that on a case-by-case basis. Like standard Google search terms, "inurl:" only searches on whole words (but doesn't seem to allow word stems), and you can only use a single word at a time in any given "inurl:" statement.
You can use multiple "inurl:" statements (one for each word) in your search, which are automatically combined with a logical AND. You can also use "-inurl:" to exclude specific URL keywords from any given search. Finally, you can combine "site:", "inurl:" and stand-alone keywords to target indexed pages by URL and content keywords in one statement.
Do you like this post? Yes No
-
An Initial Review of Boudica, the Social News Site for Women
Posted by rebecca Danny Sullivan's lovely wife, Lorna Harris (who once lent me a hat and gloves when Danny took me to see Stonehenge on an especially cold, windy day), recently created Boudica, a social news site for women.
The site is pretty new and is currently in Beta testing mode, but I thought I'd give an initial review of things thus far. I'll start with the following caveat: I'm not an especially girly female. I love Digg and reddit and don't get offended by the "omg hot girls" content that frequents more male-dominated social media sites. However, having said that, though I'm not fawning over the latest fashion trends or counting down the days until the next Matthew McConaughey shirtless pec-baring chick flick rom com, I am still female, and thus I was intrigued by Boudica and wanted to take a peek under the hood and see what sort of content it has, the community it's building, etc. I don't intend this to be a scathing review of a site that's "too girly" for my tomboy tastes. I'll aim to be fair yet straightforward, and keep in mind that the following are nothing more than my personal opinion.
What I Like About Boudica
- The community doesn't seem stereotypically "girly." Thankfully, unlike this satirical glimpse of what the Internet would look like if it were "ruled by females," Boudica features stories about women on the web, lists of "super foods," crazy fad diets, travel tips, gadgets, geek stuff, and more. In other words, you'll find a lot of stories that frequent Digg and other social news sites. Sure, there are submissions about Sex and the City, body figures, and Desperate Housewives dinnerware, but there's also content that I find appealing.
- I can discover articles and stories that may slip through the cracks on other social news sites. There was a submission that linked to a study claiming that gay men have similar brains to straight women. It's an interesting article, and I hadn't seen it on other social news sites. While I saw some submissions that were already prevalent on Digg, del.icio.us, etc, I did find some interesting submissions that I'd not seen before.
- The site transcends beyond social news and focuses on the community. Though I haven't delved too deeply into Boudica, there's plenty for me to do here. I can send private messages to other members, post a blog entry, submit stories to the social news section, scrawl a quick message on the "Scrawl Wall," and interact with members beyond simply adding them to my friends list. The various features are a nice change of pace from other social news sites--Boudica encourages participation and discussion, and it seems to reward/appreciate users who put a lot of time and effort into using the site.
- I like the marketing potential. Women-oriented sites can craft link bait and interesting articles/blog posts that appeal to women rather than trying to figure out a way to put a "techy/young male" spin on a story in hopes of getting a piece on Digg or reddit or Propeller. If Boudica gains in popularity, it can be a great marketing resource for sites that produce content/offer products that are more female-centric.
What I Don't Like About Boudica
- Purple and pink aren't really my cup of tea. Does "female" always have to equate to "pink and purple"? I like bright greens and oranges and other "web 2.0" colors--I'd love to see Boudica have a hip, cool design that's fresh and clean but doesn't feel blatantly "feminine." Also, from a usability perspective, the pink links are a bit light and can be difficult to read.
- I don't understand the top-level navigation. At this point I can't tell the difference between the News, Arts, House, Time-Off, and Talk categories. They all seem to list submissions. "News" and "Arts" could be different subject categories, but what's "House"? Are they stories that deal with home matters? That are appealing to housewives? Is it a category entirely devoted to Hugh Laurie? Is it mighty migh-tay, just lettin' it all hang out? The same goes for "Time-Off" and "Talk"--I don't understand what they signify. I think a brief but clear explanation for each category would be useful (and perhaps it's necessary to re-name the categories with something more intuitive).
- The site lists users by "karma" but doesn't explain what "karma" is. I can imagine that Karma is like your popularity or signifies the strength of your account, but it's never explicitly defined on the site. Why is karma good? How do you get more karma? Can you lose karma? What's the benefit of increasing your karma points? Do you get a nifty badge or title that you can display? This is another feature that I'd like to see fleshed out a bit more.
- The site layout is a bit too cluttered for my taste. One one page I can see submitted stories, my account information, a list of recent blog posts, a list of people on my friends list, a list of the best karma users, my friends' recent blog posts, an invitation to invite a friend to use Boudica, a tag cloud, the best published "scoops," the best upcoming "scoops," and the Scrawl Wall. It's a bit of an information overload. I don't really need to see my list of friends--that could be something I can click on and see within my profile. Recent blog posts/friends' blog posts could maybe go under a "Blog" section that gets added to the top-level navigation. I think the Scrawl Wall is cute, but it could move further down the page so that more important information (like upcoming and published scoops) can get moved further up. It seems like a lot of the information featured on the page can be better placed elsewhere on the site.
Obviously, a lot of my gripes can easily be due to the fact that Boudica's still in Beta and is working out the kinks. For example, I like the tag cloud, but it's not working properly right now (I clicked on "book review," which seemed like a popular tag based on its font size, but no stories pulled up). I certainly don't fault the site for any of the problems I've identified--I've no doubt that Boudica will strive to improve user experience and functionality based on its Beta feedback.
That being said, overall I have to say that Boudica is a pretty interesting site. I'm really curious to see how popular it will become, not just among female SEOs and marketers, but what its adoption rate will be for other women (such as people like my sister, a teacher with two kids who casually uses the Internet but isn't uber-net savvy, yet sends me interesting stories and photos she comes across every now and then). I hope the site gains in popularity--if anything, it'd be a fascinating ethnographic study to see what sort of women frequent the site and what information they think is interesting. And, of course, as I said, the marketing potential is huge (hey, you can take the girliness out of the marketer...). Obviously I'll keep hanging around Digg, reddit, Propeller, Mixx, Yahoo! Buzz, del.icio.us, StumbleUpon, and other social news and social media sites, but alhough I'll certainly keep playing with the boys in their treehouse, I'm happy to sit down for a tea party with the girls every now and then, too. :)
Do you like this post? Yes No
-
Capitalising On The Ultimate Form Of Duplicate Content
Posted by Jane Copland The first time I ever accessed the Internet was from my mother's work computer in late 1995. I was eleven years old and her homepage was set to Yahoo. I can't really remember what it looked like, but Googling (oh, I hate the irony too) "Yahoo in 1995" produced a post by John Battelle with a magnificent screen cap of the portal in the mid-90s. This was thirteen years ago (so, over half my lifetime), and my memory might not be serving me very well, but I'm fairly sure that the first thing I ever searched for was song lyrics. Probably to a very bad 1995 song. My father wanted to try it next and he searched for the lyrics to "Flower of Scotland." That, I remember.
Today, searching for lyrics is a horrendous task. Most top-ranked lyrics websites look like MySpace threw up on GeoCities and, if I dare to click on a result, inundate my computer with pop-up advertising. Earlier today, I actually stumbled on an instance of a robotic voice congratulating me for having won two iPod nanos. To get a coherent result and not be presented with the "Are You Stupid?" test, you have to memorise which sites are worthwhile to click on.
How do search engines really determine which sites should rank well for song lyrics-related material? This niche seems to be relatively competitive, with advertising being the business model of choice. The first big problem is certainly duplicate content. This is an especially important question when it comes to lyrics because of people's tendency to take a sample of a song they've heard and search for it without knowing the song's name. If there are thousands of instances of the same song present online, how does a site make sure its version is ranked?
The suggestions Google shows for searches beginning with "lyrics" is a good place to start when analysing what search engines value for these types of searches.
Currently popular music obviously dominates. Choosing the search "lyrics to take a bow," you'll see that Google presents both results for a currently popular song with that name by Rihanna, as well as a track from 2007 by Leona Lewis and a fourteen-year-old song by Madonna. Edit: two YouTube videos have made it into the mix in the last twenty-four hours, taking out the Leona Lewis song.
The top three results, plus results five, six, seven and nine are all for the same Rihanna track. In the pages' inlinks, I've included internal links, as some of these sites do interesting things with their internal link structure. When you look at the links for the LyricsMode.com page, you'll see that tens of thousands of them appear to come from pages like this, which are results pages for failed queries. Instead of displaying no content, the site shows the top 100 most popular songs at the given moment. Given that the page has only 29 links from external sources, I have to believe that its internal work is quite important here.
1) http://www.metrolyrics.com/take-a-bow-lyrics-rihanna.html
663 inlinks - PR3
2) http://www.completealbumlyrics.com/lyric/133088/Rihanna+-+Take+A+Bow.html
72 inlinks - PR2
3) http://justjared.buzznet.com/2008/03/14/rihanna-take-a-bow-lyrics/
109 inlinks - PR 6
5) http://www.lyricsmode.com/lyrics/r/rihanna/take_a_bow.html
68,982 inlinks - PR0
6) http://www.lyricstop.com/t/takeabow-rihanna.html
6 inlinks - PR0
7) http://www.musicloversgroup.com/rihanna-take-a-bow-video-and-lyrics/
415 inlinks - PR0
9) http://www.celebridiot.com/2008/04/25/rihanna-take-a-bow-video-and-lyrics/
130 inlinks - unranked
For comparison's sake, here are the links and PageRanks for the domains:
http://www.metrolyrics.com/ - 1,879,225 inlinks - PR5
http://www.completealbumlyrics.com/ - 39,336 inlinks - PR6
http://justjared.buzznet.com/ - 447,097 inlinks - PR6
http://www.lyricsmode.com - 906,098 inlinks - PR5
http://www.lyricstop.com/ - 11,433 inlinks - PR5
http://www.musicloversgroup.com/ - 59,875 inlinks - PR4
http://www.celebridiot.com/ - 526,323 inlinks - PR4
On the surface, this seems totally unexplainable. Aside from a manual tweak which somehow acknowledges that lyrics are inherently duplicated, how do search engines justify ranking the same content over and over again?
Or is this the result of literally everything related to this query being duplicate content? If search engines filter duplicate content, simply lowering results that are duplicated, then surely it stands to reason that if all the results are duplicates, then there is nothing else to be shown above affected results. However, you'd think that adding your own content and hiding the lyrics with something like an iframe, but still optimising for lyrics searches, would be beneficial. Or would this be considered too manipulative? Obviously, this would negate searches where people type in snippets of songs they've heard and want to find. For this, could you pick out which parts of songs people are most likely to include in search queries (first words, repeated phrases, hooks, etc) and include only those as indexable content, excluding the rest with whichever technique you choose. It could certainly be done, especially with iframes, and could probably look relatively natural.
The answer in regards to the Rihanna song may well be that the content is not in fact the same. Many of these lyrics websites rely on users to provide their content, and it seems to be rare that words are actually taken from official resources. Each results' lyrics are slightly different.
The common wisdom is that duplicate content will still be singled out if a degree of similarity is detected. How similar do results have to be in order to be filtered? Also, there is unique content on each of these pages, the easiest and most common being user comments about the song. How much of the content has to be duplicated, and should it make a difference that the original comments are virtually hidden whilst the lyrics are front-and-centre?
If having only ever-so-slight unique content is all it takes, this changes our duplicate content landscape a bit. Currently, we'll give people advice such as present duplicated (or substantially similar) content in an iframe, surrounding it with unique content to prevent a page from being filtered. Is it really enough to change instances of "closing" to "closin' " and "cause" to "cuz?"
Perhaps a better indication of truly duplicate content would be a lesser-known song and one that has less room for interpretation when it comes to lyrics. For this, I chose "No Aphrodisiac" by Australian band The Whitlams. The song has two lines which could be up for interpretation as far as punctuation and spelling go. However, upon searching for "lyrics to no aphrodisiac," I see that all but one site replicates the same spelling and same punctuation.
I'd like to see what would happen if a site like Last.fm began offering lyrics. Last.fm, Pandora, and similar sites provide some of the highest quality online music content and are miles ahead of Lyricsdepot, A-Z Lyrics, and other lyrics databases. Last.fm has the web presence and the community to make such a campaign work: the main question would be whether they'd be interested in harnessing that market. For informed users, a Last.fm result would be far more satisfying than the pop-up ridden, hideous results that currently rule the SERPs.
Last and Pandora would also be optimising for a different purpose: 99.9% (I'd say 100% but someone would have to prove me wrong) of ranking lyrics websites are pushing ringtone advertisements; Last and Pandora sell premium subscriptions to their online "radio" stations. Both sites show advertising, but not nearly with the saturation of lyrics databases. I have little experience with Pandora, but Last also touts links to Amazon for users to purchase CDs and mp3s. These business models are very different and undoubtedly, very few lyrics searchers will end up converting into paying Last members. However, those who do often end up providing quite a healthy stream of income as repeat customers, and the commission earned from the Amazon links probably doesn't go astray either.
Given the duplicate content and the overall horrifying quality of lyrics sites, I wonder how difficult it would be to rank well for these searches. Some of these sites' link profiles are quite impressive, but if search engines' goal is to provide the highest quality content to users, they would surely love to see a high-quality competitor take hold of the niche, whether that competitor was selling premium content or making its money through advertising.
As an aside, I have always found LyricsMode to be a lot better than most of these sites and it's good to see their rankings steadily improving. I do believe, however, that there's plenty of room for improvement in this lucrative market and if someone dares to make vast improvements, the rest of the market will follow suit.
Do you like this post? Yes No
-
White Hat Cloaking: It Exists. It's Permitted. It's Useful.
Posted by randfish I'll begin with a quote from Google's Guidelines on Cloaking:
Serving up different results based on user agent may cause your site to be perceived as deceptive and removed from the Google index.
There are two critical pieces in that sentence - "may" and "user agent." Now, it's true that if you cloak in the wrong ways, with the wrong intent, Google (and the other search engines) "may" remove you from their index, and if you do it egregiously, they certainly will. But, in many cases, it's the right thing to do, both from a user experience perspective and from an engine's.
To start, let me list a number of web properties that currently cloak without penalty or retribution.
- Google - Search for "google toolbar" or "google translate" or "adwords" or any number of Google properties and note how the URL you see in the search results and the one you land on almost never match. What's more, on many of these pages, whether you're logged in or not, you might see some different content to what's in the cache.
- NYTimes.com - The interstitial ads, the request to login/create an account after 5 clicks, and the archive inclusion are all showing different content to engines vs. humans.
- Forbes.com - Even the home page can't be reached without first viewing a full page interstitial ad, and comparing Google's "cached text" of most pages to the components that humans see is vastly different.
- Wine.com - In addition to some redirection based on your path, there's the state overlay forcing you to select a shipping location prior to seeing any prices (or any pages). That's a form the engines don't have to fill out.
- WebmasterWorld.com - Pioneers of the now permissible and tolerated "first click free," Googlebot (and only GGbot from the right set of IP addresses) is allowed access to thousands of clicks without any registration.
- Yelp.com - Geotargeting through cookies based on location; a very, very popular form of local targeting that hundreds, if not thousands of sites use.
- Amazon.com - In addition to the cloaking issues that were brought up on the product pages at SMX Advanced, Amazon does lots of fun things with their buybox.amazon.com subdomain and with the navigation paths & suggested products if your browser accepts cookies.
- iPerceptions.com - The site itself doesn't cloak, but their pop-up overlay is only seen by cookied humans, and appears on hundreds of sites (not to mention it's a project of one of Google's staffers).
- InformationWeek.com - If you surf as Googlebot, you'll get a much more streamlined, less ad-intensive, interstitial free browsing experience.
- ComputerWorld.com - Interstitials, pop-ups, and even some strange javascript await the non-bot surfers.
- ATT.com - Everyone who hits the URL gets a unique landing page with different links and content.
- Salon.com - No need for an ad sponsored "site pass" if you're Googlebot :)
- CareerBuilder.com - The URLs you and I see are entirely different than the ones the bots get.
- CNet.com - You can't even reach the homepage as a human without seeing the latest digital camera ad overlay.
- Scribd.com - The documents we see look pretty different (in format and accessibility) than the HTML text that's there for the search engines.
- Trulia.com - As was just documented this past week, they're doing some interesting re-directs on partner pages and their own site.
- Nike.com - The 1.5 million URLs you see in Google's index don't actually exist if you've got Flash enabled.
- Wall Street Journal - Simply switching your user-agent to Googlebot gets you past all those pesky "pay to access" breaks after the first paragraph of the article.
This list could go on for hundreds more results, but the message should be clear. Cloaking isn't always evil, it won't always get you banned, and you can do some pretty smart things for it, so long as you're either:
A) A big brand that Google's not going to get angry with for more than a day or two if you step over the line, OR
B) Doing the cloaking in a completely white hat way with a positive intent for users and engines.
Here's a visual interpretation of my personal cloaking scale:
Let's run through some examples of each:
Pearly White - On SEOmoz, we have PRO content like our Q+A pages, link directory, PRO Guides, etc. These are available only to PRO members, so we show a snippet to search engines and non-PRO members, and the full version to folks who are logged into a PRO account. Technically, it's showing search engines and some users different things, but it's based on the cookie and it's done in exactly the type of way engines would want. Conceptually, we could participate in Google News's first-click free program and get all of that content into the engine, but haven't done so to date.
Near White - Craigslist.org does some automatic geo-targeting to help determine where a visitor is coming from and what city's page they'd want to see. Google reps have said publicly that they're OK with this so long as Craigslist treats search engine bots the same way. But, of course, they don't. Bots get redirected to a page that I can only see in Google's cache (or if I switch my user agent). It makes sense, though - the engines shouldn't be dropped onto a geo-targeted page; they should be treated like a user coming from everywhere (or nowhere, depending on your philosophical interpretation of Zen and the art of IP geo-location). Despite going against a guideline, it's so extremely close to white hat, particularly from an intention and functionality point-of-view, that there's almost no risk of problems.
Light Gray - I don't particularly want to "out" anyone who's doing this now, so let me instead offer an example of when and where light gray would happen (if you're really diligent, you can see a couple of the sites above engaging in this type of behavior). Imagine you've got a site with lots of paginated articles on it. The articles are long - thousands of words, and even from a user experience point-of-view, the breakup of the pages is valuable. But, each page is getting linked to separately, there's a "view on one page" URL, a "print version" URL, and an "email a friend" URL that are all getting indexed. Often, when an article's interesting, folks will pick it up on services like Reddit and link to the print-only version, or to an interior page of the article in the paginated version. The engines are dealing with duplicate content out the wazoo, so the site detects for engines and 301s all the different versions of the article back to the original, view on one page source, but drops visitors who click that SERP to the article homepage in the paginated version.
Once again, the site is technically violating guidelines (and a little more so than in the near-white example), but it's still well-intentioned, and it really, really helps engines like MSN & Ask.com, who don't do a terrific job with duplicate content detection and canonicalization (and, to be fair, even Yahoo! and Google get stuck on this quite a bit). So - good intentions + positive user experience that meets expectations + use of a proclaimed shady tactic = light gray. Most of your big brand sites can get away with this ad infinitum.
Dark Gray - Again, I'll give a hypothetical rather than call someone out. There are many folks who participate in affiliate programs, and the vast majority of these send their links through a redirect in Javascript, both to capture the click for their tracking purposers and to stop link juice from passing. Some savvier site owners have realized how valuable that affiliate link juice can be and have set up their own affiliate systems that do pass link juice, often by collecting links to unique pages, then 301'ing those for bots, passing the benefit of the links on to pages on their domain where they need external links to rank. The more crafty ones even sell or divide a share of this link juice to their partners or the highest bidder. This doesn't necessarily affect visitors who come seeking what the affiliate's linked to, but it can create some artificial ranking boosts, as the engines don't want to count affiliate links in the first place, and certainly don't want them helping pages they never intended to receive their traffic.
Solid Black - Since I found some pure spam that does this, I thought I'd share. I recently performed a search at Google for inurl:sitemap.xml, hoping to get an estimate of how many sites use sitemaps. In the 9th position, I found the odd URL - www.acta-endo.ro/new/viagra/sitemap.xml.html, which redirects humans to a page on pharmaceuticals. Any time a search result misleadingly takes you to content it not only doesn't show the engine, but isn't relevant to your search query, I consider it solid black.
Now for a bit of honesty - we've recommended pearly white, near white, and yes, even light gray to our clients in the past and we'll continue to do so in the future when and where it makes sense. Search engine reps may decry it publicly, but the engines all permit some forms of cloaking (usually at least up to light gray) and even encourage it from brands/sites where it provides a better, more accessible experience.
The lesson here is don't be scared off a tactic just because you hear it might be black hat or gray hat. Do your own research, form your own opinions, test on non-client sites, and do what makes the most sense for your business and your client. The only thing we have to fear is fear itself (and overzealous banning, but that's pretty rare). :-)
p.s. The takeaway from this post should not be "cloak your site." I'm merely suggesting that inflexible, pure black-and-white positions on cloaking deserve potential re-thinking.
Do you like this post? Yes No
-
Margae vs. Clear Link: The Dangers of Doing SEO By Commission and Not Having Good Contract Discipline
Posted by Sarah Bird, Esquire May It Please the Mozzers,
For Legal Monday, I'm going to spotlight a case dealing with SEO, affiliate marketing, commissions-based SEO, and the importance of having a clear, written contract.
The Parties
In this Utah case, Margae Inc., an internet marketing company with an odd homepage, is suing Clear Link, another internet marketing company.
Clear Link is a big fish providing and managing affiliate marketing services for some hot properties, such as Direct TV, ADT, and Hughes Net.
Margae is a smaller company that claims expertise in affiliate marketing and SEO. It agreed to provide affiliate and SEO services for Clear Link.
Unfortunately, the parties disagree about the terms of that agreement.
The Facts, In Brief
Margae approached Clear Link about doing some affiliate and SEO work for Clear Link back in 2006. The parties talked and agreed that Margae would get commissions both on its affiliate sites (like any other Clear Link affiliate) and that Margae would get commissions on sales from the Clear Link properties that it optimized.
It sounds like the parties reached an agreement, right? Well, they did on the big picture stuff, but it's not clear about the details. And the devil is in the details.
Clear Link's version of the story is that it asked Margae to enter into its online "Partnership Agreement" to govern the parties' entire relationship, affiliate and SEO.
Margae's version of the story is that the Partnership Agreement covered the affiliate stuff, but that the parties had a separate oral agreement for SEO services. In fact, the president of Margae said in a sworn statement that he very specifically didn't enter into a written contract for the SEO part because he had been screwed by written contracts in the past (I'm paraphrasing there).
The parties' different understandings are in dispute. However, the parties agree that Margae entered into the Partnership Agreement and everything went quite well for about a year. Clear Link gave Margae access to its websites and Margae optimized them and received commissions. Margae also received commissions on its own affiliate sites. Everyone was making money and people were happy.
Then some time in 2007, a dispute developed between the parties. Apparently Margae was also an affiliate for one of Clear Link's competitors. According to Margae, Clear Link asked it to give up the relationship and when it refused, they terminated Margae's access to the Clear Link properties and stopped paying commission on all sites.
Margae is suing Clear Link to force it to pay any commissions earned on its affiliate sites and on the Clear Link properties it optimized.
Clear Link's position is that it can terminate affiliates whenever it wants pursuant to the partnership agreement and that it doesn't owe Margae anything more for optimizing its properties. It also argues that Margae doesn't have the legal right to bring this case because the Partnership Agreement contains a mandatory arbitration clause.
The mandatory arbitration clause argument came as a surprise to Margae because it was added to the Partnership Agreement after Margae agreed to it. Of course, the Partnership Agreement gives Clear Link the right to change the agreement without notice to affiliates.
Will Margae be able to convince a Court that the SEO services were outside of the Partnership Agreement? Will the Court enforce the mandatory arbitration clause even though it was added in after Margae entered into the agreement?
What Was the Court Was Trying to Decide in This Stage of the Case?
At this stage of the case, Clear Link asked the Court to rule that the Partnership Agreement covered the parties' entire relationship and that the parties must arbitrate their dispute under the terms of the modified electronic agreement.
Margae asked the Court to rule that the Partnership Agreement only covered the parties' affiliate marketing relationship and that there was a separate, oral agreement regarding SEO services.
Oddly, Margae wanted the same results as Clear Link; It wanted the Court to order the parties to arbitrate their dispute. However, Margae wanted the Court to reach the conclusion by different means. Margae did not want the Court to decide that the Partnership Agreement governed the parties' entire relationship (SEO and affiliate marketing). Instead, Margae asked the Court to require arbitration 'just because' Clear Link asked for it and arbitration is generally a good idea.
Why does Margae care whether the Court uses the Partnership Agreement as the reason for granting arbitration? Because if the Court rules that the Partnership Agreement is enforceable for the parties' entire relationship, then Margae could get screwed out of its commissions for the SEO work it performed on the Clear Link properties. The Partnership Agreement did not have any protection for Margae's proprietary SEO strategies and allows Clear Link to terminate business relationships at any time.
The Court's Decision
The Court agreed with Margae and found that the electronic Partnership Agreement covered the affiliate marketing services, but not the SEO services. The Court was convinced that if both parties truly wanted the Partnership Agreement to cover the SEO services, then it would have included language about website ownership, and protection of proprietary information, and some non-competition language.
It's a great ruling for Margae. Without the written agreement to rely on, Clear Link has to argue why it should be able to use Margae's SEO strategies on its site without paying Margae commission like it did in the past. In some ways, the litigation is just heating up because now the parties have to fight over the terms of the oral agreement.
However, I feel pretty confident that the Court's decision will give Margae the leverage it needs to negotiate a settlement for unpaid commissions. And both parties will start to find settlement very appealing after getting their attorneys' invoices for the litigation to date.
What Can We Learn?
First, SEO based on commissions is very risky. Yes, the potential reward is high, but the risk that your merchant will bail out on you is also high. The risk averse should not go this route.
Second, while written contracts can be scary, not having one is scarier. If Margae had a written contract covering the SEO services it was providing for Clear Link properties, it wouldn't have spent thousands of dollars just trying to prove that a contract existed. If the parties don't settle, then Margae will have to spend thousands of more dollars trying to prove what the terms of the oral agreement were.
Third, if you're drafting a contract for SEO based on commission, pay close attention to (1) your remedies and (2) the term of the contract. You need to make the merchant commit to provide you commissions for a set length of time. There should also be penalties in the event that the merchant terminates the contract early. For example, reserve the right to remove your proprietary SEO techniques and strategies in the event of contract termination.
Fourth, consider asking for a combination of a modest retainer up front and ongoing commissions. This will act as a hedge against risk and more accurately reflect the investment of time and resources you put into the project.
Fifth, don't agree to do SEO by commission unless you have tremendous trust in the honesty and the longevity of the merchant.
Lastly, if you're an online business, the courts are much more likely to hold you to your electronic contracts than a regular, unsophisticated consumer. Thus, read your contracts before you sign them and check for updates at least quarterly.
I'll keep you updated as the case develops.
Best Regards,
Sarah
Hat tip to Eric Goldman for bringing this case to my attention.
Do you like this post? Yes No
-
Tips For Understanding Data: Regression Analysis
Posted by Nick Gerner Note: This post does not have a great deal of SEO advice per se. But I will give some insight into some of the technical details of how we come up with PageStrength scores and give a sneak preview of a product we'll be launching soon.
Update: The tool mentioned below is probably more the work of the rest of the dev staff: Jeff, Mel, Mike, and Timmy. They put in a lot of long hours while I was doing stuff like this post.
Here at the mozPlex we've been hard at work on some new tools (see below for exclusive early preview screenshots!). One of the things we try to do is incorporate some of the great advice that's already out there. At the latest SMX Advanced I heard from SEOs and Search Engine reps alike that we should focus on data, and for goodness sake not just to check your rank! So we've been crunching some numbers... a lot of numbers. And all that data can start to get a little confusing. We've employed a few techniques, including regression analysis, to help us make sense of our data. I actually whipped up an online regression tool to help out.
The particular issue we're trying to understand is illustrated in the image below:
You can see that we're pulling some great data together from some very authoritative sources. However, if you notice the numbers along the right-hand side, you can see that there's a lot of different scales, and if you know anything about any of these metrics you know that the difference between 5 url mentions and 50 is quite a lot different than the difference between 20,005 and 20,050.
Suppose we're trying to understand the importance of the number of domain mentions as reported by Google. Likely, more domain mentions across the web means your domain has a greater ranking strength and influence. But how can we make that more precise? Is 100 mentions good? Is jumping from 1000 to 1100 a big jump? What should be 10% and what should be 90%? If you're a savvy SEO (and I actually know a few around here and abroad) you can come up with some examples:
| Domain Mentions |
Value |
| 1890000 |
100 |
| 1280000 |
100 |
| 866000 |
100 |
| 659000 |
96 |
| 584000 |
94 |
| 247000 |
80 |
| 115000 |
65 |
| 32500 |
45 |
| 13400 |
30 |
| 11300 |
28 |
| 6590 |
15 |
| 218 |
5 |
| 4 |
1 |
I've hidden the sites I got these from to protect the innocent ;) But you can get the idea: get some real-world examples for whom you've got some idea of how they should be labeled. Here we've said, all else being equal, 866,000 or more domain mentions is about as strong as can be. We've also given some idea of where the rest of the scores lie. The bigger question is where should we fit new data that we haven't seen before? That's where the regression analysis comes in.
The idea is to come up with some equation, a model, that matches the pattern we observe -- in this case, what you might intuitively believe as a smart SEO. You can check out what my tool will suggest for this data.
You'll notice that, in addition to the specific model I recommend, I also include a couple of graphs. If you've done this before, you know how important it can be to get a feel for what your data looks like and how your fitted model compares. I also include a graph of the "residuals," which are the errors in the model's estimates. For instance, if you ask for 80% at 247,000 domain mentions, and the model predicts 75.55, the residual is 4.45. It's often valuable to think about the square of the residual for statistical reasons (that's the "residual sq" column in the table). The square of the residual also helps to emphasize larger errors.
Intro stuff done, let's get advanced! For this particular data you'll notice (go ahead and open that "suggest" link above in a new tab), that our observations are capped at 100. This is an artifact of the 100 point scale, and poses some mathematical difficulties for simple modeling techniques. You'll also notice that the suggested model does quite poorly in the middle range. So here are a couple of tips:
If you have a truncated scale, like our 100 point scale here, it's valuable to let your model predict outside the range and truncate its estimates later. To do this you can, for example, drop the extra truncated observations from your data (1,890,000 -> 100 and 1,280,000 -> 100). Just delete those rows from the text box and click that "fit model" button and you'll get better results.
Also, the algorithm I'm using tries to match all the data points as if they are all equally important. If you feel that some range of your data is more important, just add more observations in that range. For instance, we might want to come up with some more examples between 30% and 65%, since this range is not well fitted by our initial models, and most of our users will probably fall into this range. For example, we might add 26,900 -> 38% and 47,000 -> 50%. With these new observations the model will emphasize this range of the data.
You should also consider the other possible models, even if they are not "suggested." You might like the behavior of a power model over a logarithmic, for instance. Just click the links for the other models and see the graph of the model and its residuals.
We're using these techniques in-house to score our users' pages, domains, and blogs (check out the screenshot below!), but you can use these techniques to better understand the behavior of data. The next time someone says, "What will happen if we double factor X?", you can turn to your body of experience and mathematical model and tell them, "In general, the importance of factor X falls off on a logarithmic scale. Let me show you graphically..."
Here's just one example of how we're going to use all these nifty models:
So keep your eyes peeled for evidence that college stats still matters!
Do you like this post? Yes No
-
Space Godzilla: Lessons in Monstering
Posted by willcritchlow It's a movie theme here on SEOmoz this week. After Rebecca's post on real movies, I'm going to talk about an imaginary one. It's a movie that would go straight to DVD, but might nonetheless be compelling for those of us who are search geeks.
It's a battle between monsters - the might of Google pitched against some of the largest brands in the world. Anyone see Godzilla vs. King Kong? This is Amazilla vs. King Goog (well, Amazon et al - think any large online retailer). I wanted to do that as Googzilla vs. King Amazong but then I found a picture of lizards too good to pass up (see below).
I believe there could be a war coming and it's going to be one to watch. What does the search world look like when the carnage is over? I don't know but, in a second I'll present some scenarios. First, the background:
- Increasingly, Google is encroaching into the markets of the largest web brands. There are companies who get large proportions of their revenue through search (paid and natural) for whom Google is not just a channel, but also only one step away from a competitor. For example, they have the one-box and Google Books for book searches (e.g., for me, a search for John Steinbeck has a one-box with three Google Books listings) and the payment method.
- The official best practice for on-site tactics found in the webmaster guidelines have strayed (as Rand noted recently) away from just 'think of the users' into doing many things specifically for the search engines (Google in particular).
- There is starting to be tough talk from Google about tactics that have been in place for years and are innocuous from a user's perspective but which treat the search engines differently, such as Amazon's redirects of internal navigation information in the URL that are cloaked for Googlebot. This came out at SMX Advanced, where Matt Cutts commented and there was a bit of a song and dance about the issue. There is a lot of talk from Google, but currently, I still see Amazon ranking pretty well...
Unintended consequences of actions
Google should be pretty good at thinking through unintended consequences of its actions. Some basic game theory insights (which we know they are good at over at the big G) tell us what to expect with nofollow, for example.
Nofollow has changed the link landscape to such a degree that it is starting to have the opposite effect of the intended one - when there is less incentive to get a link, it becomes ever more editorial. The quality of outbound links from Wikipedia, for example, is now actually pretty high as the incentive to game them has decreased. I would go so far as to say that on average, external nofollow links (i.e., not those you use to point at your privacy policy) might be higher value than un-nofollowed links.
As the quality increases, isn't that a signal you'd want to use in the algorithm? But it's only useful while it's not obvious that it's being used. A dilemma, and not the first example of a situation where Google's interests are not aligned with websites'.
Other unintended consequences they need to watch out for include what happens when they step into being a publisher / content creator and compete with those who use them as a sales channel.
So what happens next?
What happens when they push the giant lizards (and its friends) too far?
There is an old saying, that you if you owe the bank a million dollars, the bank owns you, but that if you owe the bank a billion dollars, you own the bank.
Some large internet brands have the power to take on the might of Google and actually hurt it. For a start, I can't imagine a site like Amazon.com getting booted out of the index. (I'm sure someone is going to bring up BMW and other examples in the comments, but I stand by this assertion. I think it would hand too much advantage to Yahoo! and Microsoft - do you fancy explaining to your non-SEO friends and relatives that they can't find stuff on Amazon through Google?)
Are we going to see some really big brands stand up to Google? Well, Amazon still has their cloaked redirect in place (try visiting this url as a regular user and as googlebot - really - check out that page. That product is AWESOME - and the inspiration for the space Godzilla in the title of the post), and you can see that they're still ranking pretty well. Is it just posturing, or has the lizard slashed the monkey? I like to think they have been asked to remove the cloaking and have said something along the lines of "Actually, we think this is the best for our users - we're keeping it." The thing is that the redirect takes you to the same page (bar minor formatting) and is the page anyone clicking through from Google will see - doesn't seem too bad to me. The extra stuff on the end of the URL for users tells Amazon that the user clicked through from Amazon's internal search (as far as I can tell).
You can see some big brands going down this route as well with the egregious link-selling going on by some of the largest brands in the world. They are sticking it to the big G and saying, "Bring it on." I'm not going to name them, but I have seen huge multi-national brands with blatant, off-topic anchor text homepage links that are not nofollowed and without a doubt are paid-for. Google might have manually stopped those links from passing juice, but they certainly haven't taken any stronger action (leading to an interesting 'caveat emptor' point - but that might be one for another post).
We are all aware that there are blackhat tactics out there, but until recently, Google could comfortably define blackhat in such a way that it was only really practised by non-brand sites (or rogue SEOs on brand sites). Increasingly, we are reaching a stage where almost any brand site is guilty of something that is against the guidelines. In a world where everyone is guilty of something, the people in power can dole out retribution on a whim.
I'm not saying that is happening at the moment, but if I were on the board of a multi-national e-commerce site making hundreds of millions of dollars a year through our natural search channel, you can bet I would be thinking about the risks - and what happens if the big monkey comes any closer.
Crazy outcome - the one in the film
It's the final show-down. The monkey has taken it too far. The lizard is properly cross.
(Thanks to futuristmovies via flickr).
If Google stepped too far into the space of its customers and the people for whom it is two sales channels (paid and natural search), and natural search traffic started dropping, for example because there was always a one-box advertising a Google product, so:
- Amazon is upset at the book and product advertising
- Wikipedia is cross with Knol
- Ebay is disappointed because of gebay (G doesn't own an auction site, do they? Bear with me here)
- Newspapers are upset by Google News
What happens if they all get together and charge the monkey? For a short period, they:
- change robots.txt: User-agent: googlebot, disallow: /
- add an interstitial advert on all pages of Amazon, Wikipedia, Ebay, NY Times, all Yahoo! and Microsoft properties, etc. to say "Google no longer has any of these brands in its search results. We suggest you use Yahoo! or Live"
...and then the world explodes.
Most likely outcome
OK. So the big fight probably won't happen. Unfortunately, I think the most likely short-term outcome is a stand-off, in which big brands approximately toe the line, step over occasionally and are not punished, but don't push it too hard. In this situation, the small webmasters lose - they are scared of what they can and can't do when the guidelines say one thing and they can find Fortune 500 companies doing something else.
I am currently in a quandary about what to suggest to clients when it comes to cloaking, for example. We all know there are white hat uses of cloaking that shouldn't really be a problem to the search engines. But they are technically against the guidelines. I thought that was a real technicality and there was no risk if not done maliciously and the on-page copy was essentially the same, but the comments about Amazon's cloaking have made me think that isn't true.
Under this scenario, one thing to be expected is that we are going to continue to see large brands buying and selling links, and rankings being achieved using bought links with only the occasional slap-down to keep the proletariat in line.
If I were Ebay or Amazon, would I buy links? Well, put it this way, you can bet I'd sponsor an ipod competition or two...
Any resemblance to a real movie plot is entirely coincidental. I have not seen Godzilla vs. King Kong and nor am I likely to.
P.S. Tom pointed me to two interesting links - one from Ciaran on a similar subject and one about Epic 2014/2015 referenced by Ciaran.
Do you like this post? Yes No
|
|
|
|
Internet Store Scroll
|
Internet News Feeds 
