HTML Tidy

The executable, not the Ruby library.

I pass everything through HTML Tidy. It's actually extremely difficult to build everything from scratch and be compliant. There are a number of things I do which are clumsy right now. HTML Tidy is a cheat.

I build everything to be XHTML 1.0 strict.

• HTML Tidy (Source Forge)

Tested and works on Unity Linux 64bit rc1 as of 2010-04-25:

cvs -d:pserver:anonymous@tidy.cvs.sourceforge.net:/cvsroot/tidy login

press enter
cvs -z3 -d:pserver:anonymous@tidy.cvs.sourceforge.net:/cvsroot/tidy co -P tidy
cd tidy/build/gmake
make
su
smart install libxslt-proc
make install

The command I use is:

system(
  'tidy',
  '-clean', 
  '-quiet', 
  '-omit', 
  '-asxhtml', 
  '-access', 
  '-modify', 
  '--drop-empty-paras', 'true', 
  '--indent', 'true', 
  '--indent-spaces', '2', 
  '--keep-time', 'true', 
  '--wrap', '0', 
  '--force-output', 'true', 
  '--show-errors', '0', 
  '--show-warnings', 'false', 
  '--break-before-br', 'true', 
  '--tidy-mark', 'false', 
  '--output-encoding', 'utf8', 
  '--escape-cdata', 'false', 
  '--indent-cdata', 'true', 
  '--hide-comments', 'true', 
  '--join-classes', 'true', 
  '--join-styles', 'true', 
  source_file_full_path
)

For additional options, check out tidy -help-config

HTML Validator

• HTML Validator (install)
• online
• (check spiralofhope.com)

• Web Design Group (WDG) HTML Validator
• (check spiralofhope.com)

A HREF function-links

JavaScript function-links are not actually valid. I've tried all sorts of stuff, but the best I can do is to wrap such things inside of <script type="text/javascript"> so that it'll only appear when JavaScript is enabled. Example link:

<a href="javascript:toggle('styles')">

To fix the validation issue, I started doing:

<a accesskey="t" href="/javascript.html#s0" onClick="javascript:toggle('styles');return false">Styles</a>

.. this forces me to have the link though. While this isn't really what I wanted, it does give an opportunity to link to another page explaining what JavaScript would have allowed the user to do.

Using HTML with document.write

JavaScript document.write technically shouldn't have HTML opening tags within it. Tidy HTML will escape any forward slashes in opening tags, ruining the code. I've never found a way around this. If I put only ending HTML tags inside the JavaScript - which is valid - and the other text outside, I still get validation errors. I have found no way around this. Example code:

<script type="text/javascript"><!--
  var heredoc = (<r><![CDATA[
    <p>some example text</p>
  ]]></r>).toString();
  document.write(heredoc);
//--></script>
<noscript>
</noscript>

Notes on specific robots

Internet Archive

• http://archive.org
• info: http://www.alexa.com/help/webmasters
• User-agent: ia_archiver

Google

• http://google.com, http://www.google.com/imghp, http://video.google.com/, etc: http://www.google.com/intl/en/options/
• http://search.aol.com/, http://search.aol.com/aol/imagehome, http://video.aol.com/
• info: http://en.wikipedia.org/wiki/Googlebot
• I can't find direct information, try around here: [5]
• submit: http://www.google.com/addurl/
• User-agent: Googlebot

Microsoft

• http://www.bing.com / http://live.com
• http://yahoo.com
• http://altavista.com
• info: http://en.wikipedia.org/wiki/Msnbot
  • I can't even give a direct URL to official information on it.
• submit: http://www.bing.com/webmaster/SubmitSitePage.aspx
• User-agent: msnbot

Yahoo (Switching to Microsoft's bing.com engine: [6] [7])

• http://yahoo.com
• http://altavista.com
• info: http://help.yahoo.com/l/us/yahoo/search/webcrawler/index.html
• submit (requires registration): http://search.yahoo.com/info/submit.html
• Yahoo directory submit: https://ecom.yahoo.com/dir/submit/intro/
• User-agent: Slurp

Cuil

• http://cuil.com
• info: http://www.cuil.com/info/webmaster_info/
• submit: http://www.cuil.com/info/contact_us/feedback/crawl_me
• User-agent: Twiceler

Sitemap

I have a plain html sitemap file, but there is also a sitemap XML standard:

• http://en.wikipedia.org/wiki/Sitemaps
• http://www.google.com/support/webmasters/bin/topic.py?topic=8476

In robots.txt, add something like:

Sitemap: http://example.com/sitemap.xml

Multiple lines for multiple sitemaps is allowed.