After a few weeks with our new responsive/mobile site not getting indexed (despite putting all information on the main page) we decided to try something else.

Previously most of our clickable "links" which trigger the website to update the content in a Single Page Application (SPA) method were list (li) items.

<li class="menuSelect" data-id="quickkey">Get a Temporary License key (Quick Key)</li>

So, we decided to update them to hyperlinks as follows:

<li><a href="/ssi/quickkey.html" class="menuSelect" data-id="quickkey">Get a Temporary License key (Quick Key)</a></li>

There's one problem with this... now the browser will take the user to a small Server Side Include (SSI) page which contains only the main content and not the "rest" of the site.

This was easily solved with a little JavaScript:

$('body').on('click', '.menuSelect', function (event) {
  event.preventDefault();
  var name = $(event.target).attr('data-id');
  var text = $(event.target).text();

  if (! $(this).hasClass("noMenuUpdate")) {
    $("#menuNav").append("<span class='addedMenuNavBarItem'>" + 
      ">> </span><a class='addedMenuNavBarItem' data-id='" + 
      name + "'>" + text + "</a>");
  }

  loadBodyContent('/ssi/' + name + '.html');

});	

All we did was add the preventDefault() method to our click code and that stopped the navigation from happening, at least for real visitors to our site.  Our hope is that Google (and other) crawlers will follow the link and index the page and it's contents.

But that brings up another problem... when a user clicks on the link from search result list from Google, Bing, etc. they'll be shown the plain SSI page, not the real page.  (See our update to this issue here!) Instead of redirecting automatically (and possibly defeating the reason for this in the first place) we added the following JavaScript to each page (using an SSI) that had an anchor with an href:

<script>
  if (window.location != 'http://m.bvstools.com/') {
    document.write("<a href='http://m.bvstools.com'>Click Here</a> to view the full site.<br><br><br><br>");
  }
 </script>

What this does is give the user a link to click on to return to the "full" site, but should, in theory, allow the site and the important contents to be indexed.

As an example, click the following link:

http://m.bvstools.com/cgi-bin/softdesc?software=MAILTOOL

You are taken to a page that displays only the "content".  That content is meant to be placed into a container (or <div>) on the main page.  But, the JavaScript that is included in the page sees that the window location isn't the home page, so it then provides a link to the user to click to return to the proper main page and allows the SPA website to function properly.

I'm sure there are other solutions (possibly more elegant as well) but we're going to see how this helps, or doesn't help, the indexing with Google and Bing.  

So far, just a couple hours after resubmitting the site as well as a sitemap to Google, we are starting to see our results show up... so, we may be on to something here.

Well, it's official.  This new method helped the indexing of our sites tremendously!

So, it appears that (for now) the web page technology is a little ahead of the crawling technology.  Crawlers like hyperlinks, not Single Page Applications (SPAs) and dynamic content swapping.

We've now updated the application even more so instead of just viewing a plain text version of a web page with a link to the home page, we now redirect to the main page and load the appropriate content.

This was done by updating the redirect.html Server Side Include (SSI) to the following:

<script>
function getParameterByName(name) {
    name = name.replace(/[\[]/, "\\[").replace(/[\]]/, "\\]");
    var regex = new RegExp("[\\?&]" + name + "=([^&#]*)"),
        results = regex.exec(location.search);
    return results === null ? "" : decodeURIComponent(results[1].replace(/\+/g, " "));
}

function createCookie(name,value,days) {
    if (days) {
        var date = new Date();
        date.setTime(date.getTime()+(days*24*60*60*1000));
        var expires = "; expires="+date.toGMTString();
    }
    else var expires = "";
    document.cookie = name+"="+value+expires+"; path=/";
}

if (window.location != 'http://m.bvstools.com/') {
	// document.write("<a href='http://m.bvstools.com'>Click Here</a> to view the full site.<br><br><br><br>");
	var software = getParameterByName('software');
	var path = window.location.pathname;
	console.log(path);
	
	if (software != '') {
		createCookie('navigateToSoftwarePage',software,1);
		window.location = 'http://m.bvstools.com';
	} else if (path != "") {
		createCookie('navigateToPage', path, 1);
		window.location = 'http://m.bvstools.com';
	}	else {
		document.write("<a href='http://m.bvstools.com'>Click Here</a> to view the full site.<br><br><br><br>");
	} 
}

</script>

So what we are doing here is first parsing the Query String value to see if there is a software name.  If there is, we create a cookie (using a couple cookie functions) named navigateToSoftwarePage and give it the value of the software.

If there is no software name, we instead create a cookie named navigateToPage which will include the pathname property of the current window location.

Once these cookies are set we then change the window's location to the main page.

That's when we will check for cookie values and load the appropriate page:

$(window).load(function() {
	// When the page has loaded
	$(".hideMe").hide();
	$("body").fadeIn(100);
	var navigateToSoftwarePage = readCookie("navigateToSoftwarePage");
	var navigateToPage = readCookie("navigateToPage");
	
	if (navigateToSoftwarePage != null) {
		loadBodyContent('/e-rpg/softdesc?software=' + navigateToSoftwarePage);
		eraseCookie("navigateToSoftwarePage");
	}

	if (navigateToPage != null) {
		loadBodyContent(navigateToPage);
		eraseCookie("navigateToPage");
	}
});

Once our page loads we look for either of the "navigateTo..." cookies and if they exist we call our loadBodyContent() function to load the appropriate information into our web page and then remove the cookie.

See, we knew there was a better way than just displaying an ugly unformatted page... we just needed the time to dig into it.  

To see how this works now, just go to Google or Bing and use the search "site:m.bvstools.com".  This will load the pages that are indexed by the search engines.