Alex Osborne

[Alex Osborne @atosborne]

@GregWiedeman Note the definition of warc-fields is worded "Allowable fields *include* ..." not "The only allowed fields are ...". Heritrix for example outputs additional warc-fields 'sourceTag' and 'outlink' that aren't defined by the spec.

[Alex Osborne @atosborne]

@GregWiedeman @anjacks0n @eengland__ Personally I like the idea of using a specific custom type like application/vnd.myproject.crawlmeta+json that not just identifies it as json but as a specific kind of json. This reduces confusion when multiple tools are using different incompatible json formats.

[Alex Osborne @atosborne]

@anjacks0n @kristsi @GregWiedeman @eengland__ Yes, metadata records can be types other than warc-fields and you can even associate several metadata records of different types with a single capture event.

[Shawn M. Jones, PhD / @[email protected] @shawnmjones]

This #WebArchiveWednesday, @StormyArchives shares screenshots of our (not quite done) Raintale GUI. Raintale stories can take time to build. With this GUI, users will be able to log in, start building a story from memento URI-Ms, and come back later to retrieve their story.

[Alex Osborne @atosborne]

@anjacks0n Do you want to change/add fields often?

[Australian Web Archive (PANDORA) @NLAPandora]

Several APS5 level jobs going at the NLA including a Web Crawl Engineer working in a small dynamic team with big ideas. Details under Digital Branch here https://t.co/0jk7RQAhaR

[Alex Osborne @atosborne]

@yvesmaurer @anjacks0n Zip. :)

[Alex Osborne @atosborne]

@anjacks0n I like pull parsers (e.g. StAX) for this sort of stuff so I can control what I keep in memory. The older I get the more I appreciate tedious but simple over concise but complex.

[Alex Osborne @atosborne]

@anjacks0n Flexible standards aren't!

[Alex Osborne @atosborne]

@hhockx See table here: https://t.co/B3AnGmA0nR

[kbtechlab @kbtechlab]

Blog post: A light tale of bug hunting an Out Of Memory problem with SolrCloud by @TokeEskildsen https://t.co/NEG7qBLQl1

[Alex Osborne @atosborne]

@anjacks0n Wikipedia says block-level CRCs and some sort of parity. So yeah probably means it did happen in the write path not on the media. We hit 3 or 4 files like that (out of probably millons) when verifying md5sums as part of a media migration.

[Alex Osborne @atosborne]

@anjacks0n @chronodm @emgillis If cost is negligible then sure why not, but avoiding compression over corruption fears is rather low on my priorities. I say that having for real resorted to restoring corrupt files with a hex editor. Would rather have had more copies and regular verification.

[Alex Osborne @atosborne]

@RoxPoNinja Currently standard WARC per-record gzip compression. Planning to explore more aggressive compression (delta encoding, dictionaries, lossy media) when WA starts dominating our storage budget.

[Alex Osborne @atosborne]

@junklight @anjacks0n Sure and of course it depends what the job is as to which of timeout killing or missing runs is more likely to lead to data loss/corruption/user disruption.

[Alex Osborne @atosborne]

@anjacks0n @junklight I'd usually pair a cron job with a monitoring rule checking the mtime of a file that indicates it completed successfully. In the rare event it hangs I want to investigate and fix the broken code rather than automatically killing the job and having it repeatedly silently fail

[Alex Osborne @atosborne]

@huwr Buy snacks in a language you can't read well. Play guess the mystery organ. Have surprise encounters with durian ice cream.

[Alex Osborne @atosborne]

@hhockx The distinction between those two words definitely has nothing to do with whether it's digital or not.

[Daniel Gomes @dcgomes77]

What is the VALUE of cultural heritage organizations that do not provide ACCESS to citizens?
"ZERO. Openess is a pre-condition", Harry Verwayen, Europeana #iipcWAC19