 - parse_http_request should also parse protocol (it sort-of does now, but
    we need to improve that - make it complete);
  - clean up the usage of http->protocol in filters.c (CMOVPERM) and jcc.c
    (get_and_analyze).
  - incorporate UrlContent from urlstr.c into http_request
  - we REALLY need to get rid of UrlContent and incorporate it into http_request
    !!!!!!!!

 - errlog.c - figure out whether the impossibility to initialize a global FILE *
   variable to "stderr" is a bug or a feature in glibc on Linux. (on Solaris it
   works just fine...)

  - in profile.c: decide whether tfidf will be mutexed ('cause it seems it
    only make sense to wrlock it anyway, because we can't rely on the fact that
    an analyzed document is unchanged (has no new words) vs. its old version.
    maybe if we start using cache and if-changed-since, we can keep the
    rwlock and read-only access to tfidf.
 - we are redownloading the same document many times, and also we're using it to
   update the tfidf object multiple times. Can we avoid doing that, other than
   using the url (which won't work with sites whose contents change daily) ?
   (maybe we could use the "if-modified-since" things in http ??? )
   also, maybe we should not add to tfidf the objects we analyze in response
   to a request for feedback, because we've theoretically just looked at them...


 - write a TFIDF server, running as a separate process. SurfAgent proxies will
   send it tf vectors, and it will respond with tfidf vectors. Then, SurfAgents
   will be able to service single users eventually, and dealing with tfidf
   will be simpler.


 - mark function parameters that aren't changed as const blah...

 - make the return convention uniform. Functions should return the same kind of
   values on success/error. (e.g. 0 success, -1 error, ptr success, NULL error)
   or ( 0 - error, 1 - success, NULL - error, ptr - success ).

 - consider removing the jar, vanilla_wafer, wafer_list, and xtra_list  !! ?


 - test html parser on file INDEX.ANML !!! (for robustness :> :> )

 - replace struct list with StrList, ultimately to be phased out by
   vector<string> in the future C++ version.

 - clean socks4.c (gnu_regex.c is just too messed up to ever get cleaned up...)

- also, figure out how to make linux prefer writers (it should, but it doesn't)
  (there's a bug in glibc, when it'll be fixed it will work :) 
  

  - check all calls to malloc() or zalloc() for errors, and make them fperror
    and exit... (basically add the error logging to logfp and fperror to
    logfp... And they must all exit on allocation errors !!!
  - Also - freez involves a test and a free - sometimes we don't need it, we
    could only do free...


  - check all the headers included - should include the minimum required...
    I have _started_ doing this, but after so much moving stuff around, I guess
    it needs re-doing...

================= For stop/stem/strlist/wordcount/tfidf ====================

#### If we're messing with the parser, we could get the alternate text found in
      images and retrieve it as words ...
- htmparse should retrieve the alt="" field in img's as words in the text.

* error checks should either exit() or their results should be interpreted
  by the calling party in a meaningful way... (mostly done - GLS )


--- we can duplicate a tfidf vector using sum-vectors with NULL (will it be
    efficient ? Do we need to ?

-*-*-* it looks more and more like the tfidf object can be adequately protected
       using just a simple mutex lock, no rw stuff needed...

----- add the stemming back by adding it to ParseHtml().

-- urlstr.c and urlstr.h
   Does the call to atoip() in urlparse() unnecessarily duplicate any other DNS
   call ?
   We should make compress_url more efficient (i.e. scan for the presence of
   /../ or /./ or // sequences or something, and if not present just return
   the whole thing...
   When comparing protocols, should we report that http and https are the same ?
   We probably will have to treat more protocols, such as ftp...
   (look in all functions for this, including parse_url() and is_qualified()...)
   Generally, we should hide the things that are not used from outside in the
   urlstr module... (like maybe the data structure, or some functions, etc.)

  -  parse_http_request() in parsers.c can be changed to make use of the
     splitter routine from above...
