Tokenizer initial commit

[SVN r10319]

.gitmodules

@@ -94,6 +94,10 @@
 	path = libs/pool
 	url = ../pool.git
 	fetchRecurseSubmodules = on-demand
+[submodule "tokenizer"]
+	path = libs/tokenizer
+	url = ../tokenizer.git
+	fetchRecurseSubmodules = on-demand
 [submodule "compatibility"]
 	path = libs/compatibility
 	url = ../compatibility.git

index.htm

@@ -63,6 +63,12 @@ that the Boost libraries are suitable for eventual standardization. Indeed, the
 explicit intent is to propose many of these libraries for inclusion in the C++
 Standard Library.</p>
 <h2><a name="Latest News">Latest News</a></h2>
+<p>June ??, 2001 - Version 1.23.0
+<ul>
+  <li><a href="libs/tokenizer/index.htm">Tokenizer Library</a> added. Break of a
+    string into a series of tokens, from John Bandela.</li>
+  <li>Please update!</li>
+</ul>
 <p>May 25, 2001 - Version 1.22.0
 <ul>
   <li><a href="libs/crc/index.html">CRC Library</a> added. Compute cyclic
@@ -203,7 +209,7 @@ efficiency improvements, co-operates with user-defined types, regression tests
   <li><a href="more/faq.htm">FAQ</a>  updated.</li>
 </ul>
 <hr>
-<p>Revised <!--webbot bot="Timestamp" s-type="EDITED" s-format="%d %B %Y" startspan -->25 May 2001<!--webbot bot="Timestamp" endspan i-checksum="15112" --></p>
+<p>Revised <!--webbot bot="Timestamp" s-type="EDITED" s-format="%d %B %Y" startspan -->12 June 2001<!--webbot bot="Timestamp" endspan i-checksum="17551" --></p>
 
 </body>
 

libs/tokenizer

@@ -0,0 +1 @@
+Subproject commit 0854dc07e99cae9603d289943dff62b1333d4952