#25 On Spidering Error, Stores that URL has been handled
When running a set of configs to get erowid, if there
is an error in the config file that results in nothing
getting saved, the db still says that the URL has been
handled.
For instance, I had a config where the <locations
download="/path/to/download"> was set wrong. This
generated an error saying No Such File in the output,
but marked the starting URL as handled and now I have
to throw out the whole run because it has a couple of
key URLs marked as handled and non-existent.
I don't know if there is a more general solution for
being able to say "overwrite URL-keys in the database
on this run", that would probably be a good feature to
post-hoc fix these errors.
To fix this bug, all that is necessary is to make
sperowider aware that it failed to actually write
anything on this run and not leave the database with
bogus "done" markers.
Logged In: YES
user_id=866724
Please provide a step by step explanation of how you caused this
bug. Don't summarize, and don't worry about providing too much
detail. This will help us reproduce this bug in a controlled
fashion, which will get us 90% of the way to fixing it.
If you can't reproduce this bug reliably, then don't forget to
mention that too!
Logged In: YES
user_id=866724
Please provide a step by step explanation of how you caused this
bug. Don't summarize, and don't worry about providing too much
detail. This will help us reproduce this bug in a controlled
fashion, which will get us 90% of the way to fixing it.
If you can't reproduce this bug reliably, then don't forget to
mention that too!