V. 0.0

- SmartCookies: now if you enter $<domain> in the cookie file, the proxy stores
                the cookies itself, thus hiding the identity of the users behind
                it. This is good for sites that force you to use cookies without
                offering anything in return (like a personal account for
                ordering stuff - which is bad too, if done via cookies, but we
                have to live with that...)
- TinyGif: when a URL is on the blocklist, the proxy won't automaticaly return
           some error-url. If the tinygif option is set, the header of the docu-
           ment is downloaded and analyzed for content. If the content is an
           image, then a special GIF is send to the client instead. If it's not
           an image, the regular error-html gets sent.
- misc.c: created this new file for various odds-and-ends-like functions, like
          zalloc(), fperror(), and strsav(). ssplit() was also moved here, and
          ssplit.c was deleted.
          fperror() had a bug: it tried to freez the string generated by
          strerror. This is BAD. It is now fixed... (it was my fault, too, since
          I removed the safe_strerror function that _did_ strdup the result of
          strerror before returning it...
- urlcodec.c: cleaned up encode.c, removed the global code maps, and wrote
              wrapper functions for the encoding function. Also, the creation of
              the code maps is now done by a separate function, which is
              supposed to be called before the multi-threaded activity begins,
              from which point on, everything is thread-safe because the
              code-maps will only be read from, not written to.
              Another thing I added was checks for NULL arguments in the encode
              and decode functions. This way we avoid the whole thing barfing
              with a segfault, due to strlen( NULL ) or something similar...
-bindconn.c: bind.c and conn.c have been merged into one single file, and the
             global variables ip_addr_str and ip_addr_long have been eliminated
             (now accept sets them directly into the csp structure).




-------------------- in the ir department ------------------

- htmparse.c: replaced wordcount.c. It is an automaton that scans characters
              and figures out when stuff happens in a html document. Currently
              it inserts words into the WordVec associated to the document, and
              inserts urls and words into the LinkVec (vector of links) that
              characterizes the set of links available in the document.
              We still need to re-add stemming and stop-word removal (remove
              this line and say we've done it when we've done it... :) )
              One more thing: there were some stupid errors when accessing docs
              in foreign languages. That happenned because in ParseHtml() we
              converted the input character to int and high-ascii chars got
              converted to negative values. Converting to unsigned char took
              care of that one...
- parsers.c: we consider a document to be text/html only if it comes back with
             "http/... 200 OK" (for the purpuse of appending the feedback button
             to it. While at that, we're looking for the "200" string, and not
             the "OK" one, because some weird web servers will send stuff like
             "HTTP/1.1 200 Orl Korrect" instead of "HTTP/1.1 200 OK". Up to this
             point, the "200" code has been the single reliable indicator of
             successful download of something... (and of course, the content
             type field must be text/html :) )


V. 0.0.0

- profile.c and profile.h
    these new modules have taken over the user profile. Now, there is multi-user
    capability, and sed() takes care of getting the user name out of the client
    header.
    The user name is validated with valid_user(), which currently only attempts
    to insert new users if the name is not found in the list.
    The user profile is all contained in one directory (defaults to "profile",
    but can be explicitly named in junkbstr.ini). Under the profile directory,
    there are the saved d and df components for tfidf analysis, and directories
    named after each registered user. In each user directory there are tfidf
    vectors saved in files named after the category they pertain to. All this
    stuff is loaded in at the start of the program, by calling load_profile().
    At the end of each download for text/html, the update_profile() routine is
    called. This routine updates the tfidf object, and either builds the list
    of categories with measures of how well the current documents matches each
    of them, or it adds the document to one of the categories, and updates the
    saved files on disk.
    The list of categories currently available to the user is transmitted via
    the "feedback" button, and displayed in the feedback window...

- errlog.c and errlog.h
    All the error logging has been rewritten as a contained object, which is
    used through the printf-like, variable-argument function errlogf(). There
    is an initialization function which sets the intended detail level for
    logging, the program name (from argv[0]) and the file to which to log.

- tfidf.c
    The WordVecLoad routine was just too slow (especially for the big df vector.
    Instead of reading a word from the file and then calling WordVecInsCnt(), I
    just decided to insert at the end of a list directly, while checking that
    the alphabetical order is respected. I have no measurements, but it's much
    faster now... (i.e. it's linear now vs. maybe n^2 before...).

- miscstr.c and miscstr.h
    Are now replacing misc.c (fperror went to errlog.c/h as described above...)
    They offer strsplit(), strsav(), and zalloc().

- linkvec.c and linkvec.h
    This module builds a vector of links found on the page being downloaded.
    Each link comes with its url and a list of words containing the underlined
    link text. The learning module has been taken out for now, and only
    underlined words are added to each link entry of the vector. This module
    is intended for harvesting links off of pages visited.

#    OLD STUFF:
#    plus words from around the link found within a window of specified
#    size. The optimal window size for predicting the contents of the document
#    linked to is learned using a Monte-carlo n-armed bandit (where n is the
#    number of window sizes considered). This module could also be useful for
#    harvesting links from a hub (news site) page.
#    The learning part compares the set of words in and around the link with the
#    words found in the destination document, and uses the match value to
#    reinforce the window size used to build the links.
##### How come winval[0] gets to be NaN ?!?!?!
      Maybe this had to do with the bug in StrListDequeue, which sometimes might
      have reached for an extra element of the list after the last one. This bug
      was fixed, now we need to make sure the NaN thing stopped occurring, too..

- urlstr.c and urlstr.h
    A module performing operations on url strings. Currently only http and https
    protocols are treated. The module offers operations for comparing url's,
    qualifying relative url's using a given base, etc.

- htmparse.c
    Now finds <base href="..."> statements and changes the base in the current
    linkvec object.

- smrtcook.c and smrtcook.h
    This module is in charge of the SmartCookie list. Whether a site has in,
    out, or smart cookie access is still determined in loaders.c and parsers.c,
    but storing and retrieving SmartCookies is the job of this module (just
    to keep rwlocked stuff under tight wraps, I guess...)

- jcc.c, profile.c, profile.h
    update_profile() used to return "categ_none" to save work at the end of a
    document downloaded for the profile only, that would have been reloaded
    anyway. Now it always returns the whole categ list when LOG_FEEDBACK is
    used. At the end of chat(), the categ list is logged together with the
    url and selected category for analysis purposes. May remove after tests
    are finished. Also, update_profile now returns a similarity metric of
    the old categ. vector and the current document (only if LOG_FEEDBACK is
    selected). Look for "LOG_FEEDBACK" to remove all things related to this...


V. 0.1.0      -- file_list, trust_list, sweep() are gone !; block lists are now
                 protected by locks, removing potential conflicts between
                 threads...

- filegard.c, h:  acl.c and loaders.c are now one single file, filegard.c
    Some of the files from filters.c have been moved to filegard.c which now
    contains the rudiments of several closely related classes (block, url,
    cookies, forwarders) which are all initialized in a similar way from their
    respective files and provide lists for all threads to use in common.
    The trust-list has been considered worthless and therefore removed...

- jcc.c and jcc.h     -- sweep() is gone !!!
    the complications of cleanup of the csp structure have been greatly
    simplified. Now each thread cleans up its own mess, and we don't have any
    list of csp's that need cleaning up before each new threread is launched.
    The de-allocations left over from sweep() are now handled at the end of
    serve(). File list structures and cross-linking of blocklists have been
    eliminated. All is now handled by filegard.c/h and there is only one
    copy of each file list for the entire process. Routines in filegard.c
    make sure that the lists are up to date and re-load them if config files
    change.

- filegard.c
    the recursive cleanup routines for obsolete lists have been replaced with
    iterative inline code, which is hopefully going to be faster...

- parsers.c, jcc.c    -- better handling of blocked IMAGES
    the function client_accept() figures out when the client wants to download
    an image exclusively, and sets a boolean flag in the csp structure.
    when an URL is blocked, chat() checks if the client expects an image
    exclusively, and if so, it sends the blank gif depending on the value of
    the tinygif variable. This removes the necessity to download the header
    of the blocked URL first, in order to figure out the type.

- strlist.c strlist.h
    added StrListFind() to return the index of the first occurrence of a string,
    and StrListRemove() to delete a string given by its index.

- profile.c profile.h
    Each user now has a monitor list added to their profile. The functions added
    are get_monlist(), add_monlist() and del_monlist() to handle retrieval of
    the list in string form, addition / deletion of a URL to the monitor list.

- filters.c
    added sa_update_monitor() to handle user interaction re. the monitor list.
    The url's are currently single strings, but it is possible to use two
    fields (hostport, path) in the future, should the need arise.

- parsers.c
    extensive cleanup, and grouping of functions according to the common
    datastructure or purpose they serve: sed, http_request, struct list, and iob

- jcc.c, jcc.h, filegard.c, parsers.c		-- one flag for cookie status
    the three flags - accept_server, send_client, smart_cookie - have been
    replaced by a single flag, cookie_status, which is treated as a bit vector
    and information is now inserted by | -ing, and extracted by & -ing with the
    cookie flags given in filegard.h

-parsers.c
    fixed parse_http to skip login:password in urls of type
      proto://login:password@domain:port/path
    when parsing the domain. Previously the domain would have been filled in
    with login:password@domain as the domain, which is obviously bad. (thanks
    Stefan Waldherr - snarfed from his version of JunkBuster)

- parsers.c, filters.c, jcc.c   blocklist circumvention
    parse_http_request() will check for the string "|noblock" appended at the
    end of the path and will set the http->noblock flag to 1 if the string is
    found. process_client_data() will not check a request with the flag set
    against the blocklist, thus circumventing the block of the site. block_url()
    creates a link with the url + "|noblock" for the user to optionally use
    the above mechanism and circumvent the block of a site.

- bindconn.c
    in atoip(), there used to be a call to gethostbyname(). This caused various
    problems, like segfaults on Solaris, and timeouts on pictures (particularly
    the sashdot icons :) ). I replaced this with gethostbyname_r(), and now the
    problems seem to be gone, on both platforms. The drawback is that the
    call has different return types and an extra parameter on Linux. Also,
    the call seems to not have been intended for direct use by the programmer,
    but for being called automatically from within gethostbyname() upon
    definition of _REENTRANT. Now I had to include an extra set of compile
    flags: SOLARIS_GETHOST and LINUX_GETHOST, to generate the correct call to
    gethostbyname_r() from each of the two platforms. Another problem is that
    things are not final with this call, and they might change in subsequent
    versions of the libraries. We shall see.

- jcc.c
   added get_and_analyze(), a simplified chat() without a client, which only
   downloads and extracts a linkvec and wordvec out of a document.
   Also, the initialization of csp->linkvec and csp->hpp is now done in a
   separate function, start_parser_wordcount(), which is called from both
   chat() and get_and_analyze().

- profile.c
   the update_profile() function is now only handling profile feedback
   requests. At the end of each download, the update_download() function is
   handling the updating of the tfidf object and the creation of an option
   string for the feedback button.

- filters.c
  sa_update_categ() now calls get_and_analyze() and update_profile(), then
  returns a CMOVPERM, thus allowing chat() to be simple again -- there is no
  more csp->sa_categname to take into account.

- profile.c
   add_user() duplicates the functionality of find_user(), only with a write
   rather than a read lock. If the lookup fails, it will also create a new
   account. The rationale for using this is as follows: we can't read-lock,
   look up, and then on failure write lock and insert directly. Someone might
   have already inserted the account. That's why we'll look up before insertion
   within a write lock. It still makes sense to call find_user() and then
   on failure call add_user(), because it is much more likely that find_user()
   will succeed, and it doesn't require an exclusive write lock. The duplication
   of the lookup is a small and very infrequent price to pay.
   The function valid_user() does this, instead of write locking directly.
   The UserList structure now has a rwlock acct_lock instead of the acct_mutex;
   this is good because now we can allow multiple readers concurrently

- profile.c
   site_monitor() crawls the sites on the monlist, and updates the user's
   list of recommended sites to read based on interest level inferred from
   the profile
   get_monres() retrieves the string generated by site_monitor(), and
   passes it to sa_feedback_url() which is supposed to display the string
   in a table next to the monitor list stuff...

- rwlock.c and rwlock.h
   rwlock.c and rwlock.h are now gone, and we use the pthread_rwlock_*()
   functions. There is still a bug in glibc that prevents writer preference,
   but that's something we can't do anything about for now... Time will fix
   this one...

- errlog.c
   system call errors now use the ERN parameter when calling errlogf(). This
   causes the errno and strerror() to be used. ERR is for internal SurfAgent
   errors that don't have an errno attached to them.

* 01-11-2000

- parsers.c and parsers.h
   the public functions of parsers.c were moved from jcc.h to parsers.h . Also,
   the struct client_state and its constructor and destructor were moved to
   parsers.h and parsers.c, respectively.

- jcc.h
   the functions and variables declared here have been mostly scattered about
   into headers which belong to their respective modules. Now the thing is a lot
   cleaner, and everyone includes the header whose services they will be using.

- sed.h and sed.c
   the sed() function now has its own module

- csp.h and csp.c
   the client_state structure now has its own module

* 01-12-2000

- strlist.c
   all the functions return 0 on success and -1 on error, except the ones
   returning pointers which return ptr on success and NULL on error...
   added StrListSave, to dump the contents of a string list into a file

- profile.c
   Each user's site monitor is loaded from the profile directory. Anytime the
   monitor list changes, the changes are saved back into the user's monitor
   file. The monitor file is in the same directory as the categories, and
   is named using the string defined as SITE_MON in jcc.h. This is a reserved
   keyword, and no category can be named using it (this is checked for in
   filters.c)

* 01-17-2000

- stopword.c and stopword.h
   These files are the stopword removal module. Stop words are contained in a
   trie, and the identification of stopwords is done incrementally. Stopwords
   are simply not added to any word vector, thus reducing the amount of time
   needed to process any particular word vector.
   Initialization of the stopword module is called from htmparse.c's parser
   initialization procedure, which receives a file argument for the stoplist
   from load_profile(), which is in charge of which file is where in the profile
   directory.
   The stoplist module can be used uninitialized, case in which it does not
   identify any stopwords (i.e., it passes everything).
   As a general idea of size and speed: the size of the trie is 960 nodes for
   the 429-word stoplist of Frakes et.al. The speed is FAST. one pointer jump
   per character, done while characters are parsed from the buffer, with almost
   zero overhead...

* 01-20-2000

- profile.c
   An exploration probability (EPSILON=0.01) was added to the site_monitor()
   function. Sites that do not meet the current retrieval threshold of the
   category they're tested against are given a small chance to be included
   anyway for exploration's sake.
   Also, a learning rate (ALPHA=0.1) drags the cathegory threshold towards the
   prior similarity between the document being added and the cathegory. The idea
   is that if the article was better than the threshold, we should reduce the
   amount of garbage, and if the article was worse, then the threshold is too
   high or the category isn't focused well enough. This should make everyone
   happy... Thresholds are set to 0.5 when the proxy is started, an can't be
   saved on the disk for now. This will probably be fixed in the future...

* 01-24-2000

- profile.c
   thresholds are saved to disk together with their categories, and loaded on
   program startup.

* 01-25-2000

- tfidf.c
   fixed WordVecSimilarity(): if an argument was null, the way the math worked
   was to return NaN instead of 0.0 like it should have. This bug has now been
   fixed
- profile.c
   update_profile() now logs (#ifdef LOG_FEEDBACK) the pre and post thresholds
   of the category being updated.

* 01-27-2000

- profile.c
   load_profile() is now fixed up and does no longer use chdir() and getwd(),
   which used to genereate complaints at link-time under linux. Also there was
   a bug about not being able to find files due to relative path mixups that
   was eliminated.

* 01-28-2000

- sed.c, csp.c
   because technically a browser could use multiple "accept:" fields to signal
   what type of document it expects from the server, each csp will have the
   "accept_image_only" flag set to true, and it will be the job of the
   client_accept() handler in sed.c to turn it off whenever an accept field
   contains anything other than an image...
- filegard.c, h		show-proxy-args functionality has been restored...
   each fileguard now has a string in which it saves the config file upon
   reading it from disk. These strings are retrieved by the new function
   get_proxy_args(), which makes them available to the show_proxy_args() inter-
   ceptor from filters.c

* 01-29-2000

- profile.c
   update_profile() now conditionally (#ifdef LOG_FEEDBACK) includes a string
   parameter (url), for which it looks in the acct_monres list of the user when
   feedback occurs. This helps establish whether the document was a hit or a
   miss (whether the SiteMonitor found it or not before it was seen and liked).

* 01-31-2000

- tfidf.c
   TfIdfAnalyze() does no longer return the wv it received as an argument. The
   vector remains structurally unchanged, only count values change from integer
   count to weighted tfidf. The function now returns an integer indication of
   success/failure (0/-1)

V. 0.2.0	- first released version. Has "SurfAgent" logos and buttons.
                - there is a gpl message included with all files.

* 02-10-2000

- jcc.c
  get_and_analyze() had a bug: when there was no body, the part reading the
  header never finished (because it didn't get to read the "\n\n" sequence and
  went into an infinite loop instead. Dealing with read() == 0 in the header
  portion fixed this problem.

- errlog.c
  now there is an extra loglevel - THR - which can be used to print information
  related to threaded operation (e.g., when a lock is requested, granted, or
  released).


V. 0.2.1	- includes above bugfixes.

*06-05-2000

- jcc.[ch] is now chat.[ch], because the biggest part of the .c file was
  dedicated to chat routines (serve(), chat(), get_and_analyze() ). We will
  need to isolate the main routine and the main defines into surfmain.[ch] but
  that can wait.

*06-07-2000
- linkvec.c
  functions now check for the existence (non-NULLness) of parameter LinkVec.

*06-12-2000
- trie.[ch]
  these files now handle the trie data structure used by stopword.[ch].
- tfidf.c
  also, tfidf uses a trie for the df structure, instead of a word vector. This
  is supposed to be faster, although it's murder on the memory...

*06-14.2000
- hash.[ch], tfidf.c
  a hash table module has been added, which is used by the tfidf instead of
  the trie. The trie didn't work cause of the memory requirements...

*06-15-2000
- tfidf.[ch]
  got split into the tfidf object manipulation routines and a module:
- wordvec.[ch]
  was created to hold all the WordVec manipulation stuff

*06-19-2000
- wordvec.[ch] and all its dependencies
  the WordVec type got changed to include a length for the vector. The length
  is first computed when tf-idf analysis is performed on the vector.
  Hopefully, having the length present helps speed up the similarity computation
  slightly.
  The WordVecSum() operation now adds the two vectors without calling the
  insertion operator, hopefully also speeding things up a bit.
  WordVecSum() now changed to WordVecSumWtNrm(), and allows summation of
  vectors that can be normalized and arbitrarily weighted before the operation
  is performed.

*07-19-2000
- sed.c
  added the referrer forging option to client_referrer()
- blckstat.[ch]
  keeps block/pass statistics on web pages; adapted from a fix to JunkBuster
  and made to work with threads (current fork-based JunkBuster implementation
  does not work under unix, but SurfAgent uses PThreads all the way, whew ! :)

*07-19-2000
- filegard.c
  the suppress_message has been added. If set, the contents of the block, cookie
  and forward files is no longer displayed on the show-proxy-args page.


V. 0.2.2	- includes above bugfixes.

*07-23-2000
- profile.c
  optimized learning parameters according to the little test I ran:
  alpha = 0.2, thr_ini = 0.5, docw = 0.25

