Recent changes to bugs

List.cc:248: bad code order ?

dcb — Sat, 12 Sep 2015 19:49:53 -0000

[List.cc:248]: (error) Returning/dereferencing 'o' after it is deallocated / released

Source code is

if(action == LIST_REMOVE_DESTROY) delete o;
return List::Remove(o);

After something has been deleted, you can't use it.

local_user_urls can't locate msoffice extensions locally

Manoj Agarwal — Fri, 24 Feb 2012 09:52:29 -0000

If local_urls_only is set to true, and local_user_urls is used, then extensions that are searchable as local filesystem access are .html, .htm, .txt, .asc, .ps, .eps and .pdf. Can't we add .doc, .xls, .ppt, .docx, xlsx, pptx extensions also, so that these documents are also looked from local filesystem instead of going to http.

user_agent not used for parsing robots.txt

mikes123 — Sun, 17 Oct 2010 17:29:16 -0000

If you add an user_agent: config, that string is used for http gets. However, it is NOT used when parsing the returned robots.txt.

If I set "user_agent: agent-foo" in htdig.conf, and robots.txt to:
User-agent: agent-foo
Disallow:
User-agent: *
Disallow: /

...then htdig will http/get robots.txt using the user-agent string "agent-foo". However, it will parse robots.txt "using myname = htdig". It therefore thinks it is not allowed to dig.

I realize that robots.txt doesn't provide real security. But, this provides some means of allowing indexing only locally, since a unique user-agent string can be created.

can't search for strings with underscores

Anonymous — Tue, 20 Jul 2010 17:55:21 -0000

The underscores are stripped from all searches with no way to escape them. This makes the search useless for developers.

Cannot build on Mac OSX 10.5

Anonymous — Thu, 20 Nov 2008 15:49:46 -0000

First, configure fails with:

...
checking for fstream.h... no
configure: error: To compile ht://Dig, you will need a C++ library. Try installing libstdc++.

After searching the FAQ, I was instructed to use the following flags:

CPPFLAGS=-Wno-deprecated
CXXFLAGS=-Wno-deprecated

This allows configure to run, but then make fails with:

gcc -c -O -I. -I./../include -Wno-deprecated ../btree/bt_compare.c
cc1: warning: command line option "-Wno-deprecated" is valid for C++/ObjC++ but not for C
In file included from ../btree/bt_compare.c:54:
/usr/include/sys/types.h:152: error: two or more data types in declaration specifiers
/usr/include/sys/types.h:167: error: two or more data types in declaration specifiers
/usr/include/sys/types.h:172: error: two or more data types in declaration specifiers
/usr/include/sys/types.h:230: error: two or more data types in declaration specifiers
/usr/include/sys/types.h:235: error: two or more data types in declaration specifiers
make[1]: *** [bt_compare.o] Error 1
make: *** [all] Error 1

So, I cleared the CPPFLAG, which removes the warning above, but the rest of the errors persist.

I'm using:

OSX 10.5.5
gcc 4.0.1
htdig 3.1.6

Thanks,

Brian Janes
btjanes@rtda.com

robots.txt "allow" lines ignored

Dan Muller — Thu, 08 Nov 2007 15:45:56 -0000

Noticed this problem while using 3.16, Verified the cause by examining the code. The code for the latest beta release, 3.2.0b6, appears to still have the problem.

According to my reading of the relevant RFC, robots.txt files are supposed to be processed by applying the first allow or disallow line that applies to given page. Order matters! For instance, I was trying to use lines like this:

Allow: /index.pl?action=index
Disallow: /index.pl?action=

This should have disallowed action= queries in general, but allowed action=index queries. htDig ignores the allow line, and consequently does not index the action=index page.

Looks like a fairly extensive rewrite of robots.txt processing would be necessary to get the correct behavior.

htdig contains GPL incompatible licenses

mskibbe — Wed, 22 Aug 2007 12:54:47 -0000

Firstly, it is important to point out that htdig declared itself to be LGPL licensed in the readme, but GPL under http://www.htdig.org/ (link to "license information"). On SourceForge it is, again, declared to be LGPL. The package contains numerous GPL files. A combined product of GPL plus other licenses is a derived GPL work which should, accordingly, be licensed under the GPL. Proceeding on the premise that htdig is de facto GPL licensed, there are a number of GPL incompatible licenses in htdig:

There are a number of files in htdig containing various GPL incompatible licenses. A good example is htdig-3.2.0b6/db/parisc.gcc which contains the following license header:

* Copyright (c) 1996-1997, The University of Utah and the Computer Systems
* Laboratory at the University of Utah (CSL). All rights reserved.
*
* Permission to use, copy, modify and distribute this software is hereby
* granted provided that (1) source code retains these copyright, permission,
* and disclaimer notices, and (2) redistributions including binaries
* reproduce the notices in supporting documentation, and (3) all advertising
* materials mentioning features or use of this software display the following
* acknowledgement: ``This product includes software developed by the Computer
* Systems Laboratory at the University of Utah.''
*
* THE UNIVERSITY OF UTAH AND CSL ALLOW FREE USE OF THIS SOFTWARE IN ITS "AS
* IS" CONDITION. THE UNIVERSITY OF UTAH AND CSL DISCLAIM ANY LIABILITY OF
* ANY KIND FOR ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE.
*
* CSL requests users of this software to return to csl-dist@cs.utah.edu any
* improvements that they make and grant CSL redistribution rights.
*/

This license has a BSD 4 "Advertising Clause" as well as a request to return improvements to CSL and to grant redistribution rights to CSL for such improvements.

There are other files in htdig under other GPL incompatible licenses:

htdig-3.2.0b6/libhtdigphp/dynlib.m4 (PHP license)
htdig-3.2.0b6/htlib/mhash_md5.c (RSA license)
htdig-3.2.0b6/htlib/mhash_md5.h (RSA license)
htdig-3.2.0b6/installdir/english.aff (Advertising clause)
htdig-3.2.0b6/test/dummy.affixes (Advertising clause)
htdig-3.2.0b6/htlib/strptime.cc (BSD 4)
htdig-3.2.0b6/installdir/HtFileType-magic (BSD 4)

The first step towards resolving these GPL incompatibilities is to see if the files are actually needed to build htdig or if they are simply present. If the files are actually needed, it may be useful to see if they are actually linked in the build process.

Windows Segmentation Fault When Unexpected Data Entered

Anonymous — Fri, 04 May 2007 18:46:57 -0000

What Happened?

Running the dig on command on Windows and mis-entering data causes 'dig' to seg fault dumping information to screen

What Should Have Happened?

A graceful error message

Steps To Reproduce:

1. Download current dig Windows binary for dig from dig site (unknown which one)
2. Install per prompts (instructions)
3. cd c:\dig
4. type "dig -mx goolge.com +shrot
5. Observe application crash with debug info (see below)

---output follows ---

C:\Program Files\Windows Resource Kits\Tools>cd c:\

C:\>cd dig

C:\dig>dig -mx google.com +shrot
add 00552430 size 96 file \cvs\9_3_x\lib\isc\task.c line 1064 mctx 00553EB0
add 005524A8 size 12 file \cvs\9_3_x\lib\isc\task.c line 1080 mctx 00553EB0
add 00552548 size 104 file \cvs\9_3_x\lib\isc\task.c line 181 mctx 00553EB0
add 005525C0 size 76 file \cvs\9_3_x\lib\isc\timer.c line 766 mctx 00553EB0
add 00552618 size 32 file \cvs\9_3_x\lib\isc\heap.c line 81 mctx 00553EB0
add 005526D8 size 248 file socket.c line 2726 mctx 00553EB0
add 00A840C0 size 540 file socket.c line 563 mctx 00553EB0
add 00A88300 size 588 file c:\cvs\9_3_x\lib\isc\win32\../entropy.c line 670 mctx
00553EB0
add 00A8E810 size 704 file \cvs\9_3_x\lib\dns\openssl_link.c line 105 mctx 00552
870
add 0055F6A8 size 32 file \cvs\9_3_x\lib\dns\openssl_link.c line 105 mctx 005528
70
add 00552940 size 20 file \cvs\9_3_x\lib\dns\openssl_link.c line 105 mctx 005528
70
add 0055F6D8 size 112 file \cvs\9_3_x\lib\dns\openssl_link.c line 105 mctx 00552
870
add 0055F758 size 144 file \cvs\9_3_x\lib\dns\openssl_link.c line 105 mctx 00552
870
add 00AA2B58 size 208 file \cvs\9_3_x\lib\dns\openssl_link.c line 105 mctx 00552
870
add 0055F7F0 size 68 file \cvs\9_3_x\lib\isc\mem.c line 1395 mctx 00553EB0
add 00AAAC68 size 4464 file \cvs\9_3_x\bin\dig\dighost.c line 662 mctx 00553EB0
add 00AAFE00 size 4464 file \cvs\9_3_x\bin\dig\dighost.c line 662 mctx 00553EB0
Invalid option: +shrot
Usage: dig [@global-server] [domain] [q-type] [q-class] {q-opt}
{global-d-opt} host [@local-server] {local-d-opt}
[ host [@local-server] {local-d-opt} [...]]

Use "dig -h" (or "dig -h | more") for complete list of options

C:\dig>dig -mx google.com +short
add 00552430 size 96 file \cvs\9_3_x\lib\isc\task.c line 1064 mctx 00553EC0
add 005524A8 size 12 file \cvs\9_3_x\lib\isc\task.c line 1080 mctx 00553EC0
add 00552548 size 104 file \cvs\9_3_x\lib\isc\task.c line 181 mctx 00553EC0
add 005525C0 size 76 file \cvs\9_3_x\lib\isc\timer.c line 766 mctx 00553EC0
add 00552618 size 32 file \cvs\9_3_x\lib\isc\heap.c line 81 mctx 00553EC0
add 005526D8 size 248 file socket.c line 2726 mctx 00553EC0
add 00A840C0 size 540 file socket.c line 563 mctx 00553EC0
add 00A88300 size 588 file c:\cvs\9_3_x\lib\isc\win32\../entropy.c line 670 mctx
00553EC0
add 00A8E7E8 size 704 file \cvs\9_3_x\lib\dns\openssl_link.c line 105 mctx 00552
870
add 0055F6B8 size 32 file \cvs\9_3_x\lib\dns\openssl_link.c line 105 mctx 005528
70
add 0055F6E8 size 20 file \cvs\9_3_x\lib\dns\openssl_link.c line 105 mctx 005528
70
add 0055F708 size 112 file \cvs\9_3_x\lib\dns\openssl_link.c line 105 mctx 00552
870
add 0055F788 size 144 file \cvs\9_3_x\lib\dns\openssl_link.c line 105 mctx 00552
870
add 00AA2CE0 size 208 file \cvs\9_3_x\lib\dns\openssl_link.c line 105 mctx 00552
870
add 0055F820 size 68 file \cvs\9_3_x\lib\isc\mem.c line 1395 mctx 00553EC0
add 00AAADF0 size 4464 file \cvs\9_3_x\bin\dig\dighost.c line 662 mctx 00553EC0
add 00AAFF88 size 4464 file \cvs\9_3_x\bin\dig\dighost.c line 662 mctx 00553EC0
add 0055F8D0 size 880 file \cvs\9_3_x\bin\dig\dighost.c line 326 mctx 00553EC0
add 00AB5120 size 2064 file \cvs\9_3_x\bin\dig\dighost.c line 505 mctx 00553EC0
add 0055FC50 size 44 file \cvs\9_3_x\lib\isc\event.c line 51 mctx 00553EC0
del 0055FC50 size 44 file \cvs\9_3_x\lib\isc\event.c line 38 mctx 00553EC0
add 0055FC88 size 260 file \cvs\9_3_x\lib\dns\message.c line 683 mctx 00553EC0
add 0055FD98 size 68 file \cvs\9_3_x\lib\isc\mem.c line 1395 mctx 00553EC0
add 0055FDE8 size 68 file \cvs\9_3_x\lib\isc\mem.c line 1395 mctx 00553EC0
add 00AB9950 size 548 file \cvs\9_3_x\lib\isc\buffer.c line 379 mctx 00553EC0
add 00ABDBA0 size 2064 file \cvs\9_3_x\bin\dig\dighost.c line 505 mctx 00553EC0
add 0055FC50 size 44 file \cvs\9_3_x\lib\dns\message.c line 2266 mctx 00553EC0
add 0055FE38 size 60 file \cvs\9_3_x\lib\dns\message.c line 2303 mctx 00553EC0
add 00AD63F0 size 65535 file \cvs\9_3_x\bin\dig\dighost.c line 1779 mctx 00553EC
0
add 0055FE88 size 268 file \cvs\9_3_x\bin\dig\dighost.c line 1820 mctx 00553EC0
add 00AC63E8 size 65535 file \cvs\9_3_x\bin\dig\dighost.c line 1844 mctx 00553EC
0
add 0055FF98 size 96 file \cvs\9_3_x\lib\isc\timer.c line 341 mctx 00553EC0
add 00AEA410 size 4096 file \cvs\9_3_x\lib\isc\heap.c line 126 mctx 00553EC0
add 00AEB420 size 324 file socket.c line 1719 mctx 00553EC0
add 00AEF588 size 148 file \cvs\9_3_x\lib\isc\event.c line 51 mctx 00553EC0
add 00AF3640 size 148 file \cvs\9_3_x\lib\isc\event.c line 51 mctx 00553EC0
del 00AF3640 size 148 file \cvs\9_3_x\lib\isc\event.c line 38 mctx 00553EC0
add 00AF36E0 size 44 file \cvs\9_3_x\lib\isc\event.c line 51 mctx 00553EC0
del 00AF36E0 size 44 file \cvs\9_3_x\lib\isc\event.c line 38 mctx 00553EC0
del 0055FF98 size 96 file \cvs\9_3_x\lib\isc\timer.c line 292 mctx 00553EC0
add 0055FF98 size 96 file \cvs\9_3_x\lib\isc\timer.c line 341 mctx 00553EC0
add 00AF3640 size 148 file \cvs\9_3_x\lib\isc\event.c line 51 mctx 00553EC0
del 00AF3640 size 148 file \cvs\9_3_x\lib\isc\event.c line 38 mctx 00553EC0
add 00AF36E0 size 44 file \cvs\9_3_x\lib\isc\event.c line 51 mctx 00553EC0
del 00AF36E0 size 44 file \cvs\9_3_x\lib\isc\event.c line 38 mctx 00553EC0
del 0055FF98 size 96 file \cvs\9_3_x\lib\isc\timer.c line 292 mctx 00553EC0
add 0055FF98 size 96 file \cvs\9_3_x\lib\isc\timer.c line 341 mctx 00553EC0
add 00AF3640 size 148 file \cvs\9_3_x\lib\isc\event.c line 51 mctx 00553EC0
del 00AF3640 size 148 file \cvs\9_3_x\lib\isc\event.c line 38 mctx 00553EC0
add 00AF36E0 size 44 file \cvs\9_3_x\lib\isc\event.c line 51 mctx 00553EC0
del 00AF36E0 size 44 file \cvs\9_3_x\lib\isc\event.c line 38 mctx 00553EC0
;; connection timed out; no servers could be reached
del 0055FF98 size 96 file \cvs\9_3_x\lib\isc\timer.c line 292 mctx 00553EC0
del 00AEF588 size 148 file \cvs\9_3_x\lib\isc\event.c line 38 mctx 00553EC0
del 00AC63E8 size 65535 file \cvs\9_3_x\bin\dig\dighost.c line 1194 mctx 00553EC
0
del 0055FE88 size 268 file \cvs\9_3_x\bin\dig\dighost.c line 1197 mctx 00553EC0
del 00ABDBA0 size 2064 file \cvs\9_3_x\bin\dig\dighost.c line 1237 mctx 00553EC0

del 0055FE38 size 60 file \cvs\9_3_x\lib\dns\message.c line 430 mctx 00553EC0
del 0055FC50 size 44 file \cvs\9_3_x\lib\dns\message.c line 435 mctx 00553EC0
del 00AEB420 size 324 file socket.c line 1815 mctx 00553EC0
del 00AB9950 size 548 file \cvs\9_3_x\lib\isc\buffer.c line 410 mctx 00553EC0
del 0055FD98 size 68 file \cvs\9_3_x\lib\isc\mem.c line 1496 mctx 00553EC0
del 0055FDE8 size 68 file \cvs\9_3_x\lib\isc\mem.c line 1496 mctx 00553EC0
del 0055FC88 size 260 file \cvs\9_3_x\lib\dns\message.c line 781 mctx 00553EC0
del 00AD63F0 size 65535 file \cvs\9_3_x\bin\dig\dighost.c line 1248 mctx 00553EC
0
del 00AAFF88 size 4464 file \cvs\9_3_x\bin\dig\dighost.c line 1253 mctx 00553EC0

del 00AAADF0 size 4464 file \cvs\9_3_x\bin\dig\dig.c line 1657 mctx 00553EC0
del 00552548 size 104 file \cvs\9_3_x\lib\isc\task.c line 168 mctx 00553EC0
del 005524A8 size 12 file \cvs\9_3_x\lib\isc\task.c line 1028 mctx 00553EC0
del 00552430 size 96 file \cvs\9_3_x\lib\isc\task.c line 1033 mctx 00553EC0
del 0055F8D0 size 880 file \cvs\9_3_x\bin\dig\dighost.c line 331 mctx 00553EC0
del 00AB5120 size 2064 file \cvs\9_3_x\bin\dig\dighost.c line 568 mctx 00553EC0
del 0055F820 size 68 file \cvs\9_3_x\lib\isc\mem.c line 1496 mctx 00553EC0
del 00A840C0 size 540 file socket.c line 2835 mctx 00553EC0
del 005526D8 size 248 file socket.c line 2861 mctx 00553EC0
del 00AEA410 size 4096 file \cvs\9_3_x\lib\isc\heap.c line 111 mctx 00553EC0
del 00552618 size 32 file \cvs\9_3_x\lib\isc\heap.c line 113 mctx 00553EC0
del 005525C0 size 76 file \cvs\9_3_x\lib\isc\timer.c line 897 mctx 00553EC0
del 0055F6E8 size 20 file \cvs\9_3_x\lib\dns\openssl_link.c line 112 mctx 005528
70
del 0055F708 size 112 file \cvs\9_3_x\lib\dns\openssl_link.c line 112 mctx 00552
870
del 0055F788 size 144 file \cvs\9_3_x\lib\dns\openssl_link.c line 112 mctx 00552
870
del 00AA2CE0 size 208 file \cvs\9_3_x\lib\dns\openssl_link.c line 112 mctx 00552
870
add 00AA2DC0 size 340 file \cvs\9_3_x\lib\dns\openssl_link.c line 105 mctx 00552
870
add 00552550 size 104 file \cvs\9_3_x\lib\dns\openssl_link.c line 105 mctx 00552
870
add 005525C8 size 72 file \cvs\9_3_x\lib\dns\openssl_link.c line 105 mctx 005528
70
add 0055F6E8 size 20 file \cvs\9_3_x\lib\dns\openssl_link.c line 105 mctx 005528
70
del 00A8E7E8 size 704 file \cvs\9_3_x\lib\dns\openssl_link.c line 112 mctx 00552
870
del 0055F6B8 size 32 file \cvs\9_3_x\lib\dns\openssl_link.c line 112 mctx 005528
70
del 00A88300 size 588 file c:\cvs\9_3_x\lib\isc\win32\../entropy.c line 815 mctx
00553EC0
12: 1 gets, 0 rem
32: 1 gets, 0 rem
44: 5 gets, 0 rem
60: 1 gets, 0 rem
68: 3 gets, 0 rem
76: 1 gets, 0 rem
96: 4 gets, 0 rem
104: 1 gets, 0 rem
148: 4 gets, 0 rem
248: 1 gets, 0 rem
260: 1 gets, 0 rem
268: 1 gets, 0 rem
324: 1 gets, 0 rem
540: 1 gets, 0 rem
548: 1 gets, 0 rem
588: 1 gets, 0 rem
880: 1 gets, 0 rem
>= 1100: 7 gets, 0 rem
Dump of all outstanding memory allocations:
None.

C:\dig>win

C:\dig>ver

Microsoft Windows XP [Version 5.1.2600]

C:\dig>date
The current date is: Fri 05/04/2007
Enter the new date: (mm-dd-yy)

C:\dig>

Results not displaying in wrapper file

Anonymous — Mon, 05 Feb 2007 14:07:59 -0000

We had htdig working fine before however I wanted to use the wrapper.html to incorporate the results within the web page.

I uncommented the wrapper.html in the conf file and it worked fine, when the database reindexed itself, it picks up the amount of results, however it doesn't display them where the $(HTSEARCH_RESULTS) is.

***conf file***

database_dir: /htdig/db

common_dir: /htdig/common

# specify the header/footer/wrapper files
search_results_header: ${common_dir}/header.html
search_results_footer: ${common_dir}/footer.html
search_results_wrapper: ${common_dir}/wrapper.html
nothing_found_file: ${common_dir}/nomatch.html

It was working ok a few days ago though, even after uncommenting the wrapper line

Website underselling ht://dig

Colin Hogben — Tue, 23 Jan 2007 12:00:43 -0000

Investogating ht://dig for possible use as an intranet search, I nearly dismissed it out of hand.

The "Features and Requirements" page says:

Searching of HTML and text files
Both HTML documents and plain text files can be searched. Searching of other file types will be supported in future versions.

which translates to "searching files other than HTML and plain text is not supported in the current version".

Fortunately I did not believe this, and eventually found under the FAQ a passing reference to PDF in 1.13 and then in sections 4.8 and 4.9 the concept of an external parser or converter.

I think your Features page should say something like:

Searching of many file types
Both HTML and plain text are searched natively, and PDF, Word, Excel, Powerpoint documents via external helpers. Support for further file types can be added via a plugin system."

Incidentally, I also think your homepage should include approximately the first paragraph of the Introduction where it can be seen, BEFORE the News. Remove references to defunct search engine Infoseek etc.

See http://www.useit.com/alertbox/20020512.html