org.hd.d.pg2k.svrCore.datasource
Class ExhibitDataFileSource

java.lang.Object
  org.hd.d.pg2k.svrCore.datasource.ExhibitDataFileSource

All Implemented Interfaces:

SimpleExhibitPipelineIF, BasicVarMgrInterface, SimpleVariablePipelineIF

public final class ExhibitDataFileSource
extends java.lang.Object
implements SimpleExhibitPipelineIF

Exhibit pipeline stage that fetches its data directly from a filesystem. This emulates the way that the pre-PG2K Gallery accesses its data.

That this considers itself to be the definitive master data store for persistent system variables (particularly event histories) and will try to use the persistent data area to store them, in a robust way with a history.

Past event values returned by this routine are considered to be authoritative, and upstream consumers can cache such values knowing them to be definitive.

All requests for valid events/periods should be responded to as authoritative or not depending on the slot time as follows:

• In the future: null are returned because no data *can* exist yet.
• For the current interval: a non-authoritative, non-null response.
• For any past interval: an authoritative response.

This is assumed not to need to filter out duplicate event/var updates (nor bad timestamps) which is slow and may not even matter here. Any network connection upstream of us should do such filtering.

We don't aim to be massively efficient with this since we hope the cacheing stage that should normally be downstream of us will compensate for our main inefficiencies. We might, however, switch to some sort of non-blocking I/O in future so that failures of networked filesystems, (etc) internally won't stop Web service.

We locate configuration files and data using LocalProps.

logger

private final SimpleLoggerIF logger

Logger; never null but may be a dummy.

MINIMISE_FS_POWER

private static final boolean MINIMISE_FS_POWER

If true then when conserving power eliminate much activity that might force activity on the data area. This includes such things as checking status/flags, that might force an automount for example, using extra energy.

This may prove too conservative, eg refusing to look for new files at all even when explicitly indicated.

See Also:

Constant Field Values

TS_LSbytes_HASH

private static final int TS_LSbytes_HASH

Number of least-significant bytes of AllExhibitImmutableData timestamp to use as hash of full collection. Using a value of 1 would imply keeping about quarter-second timestamp precision (which does not exist in UNIX systems, typically). Using a value of 2 would imply about 1 minute precision. Using a value of 3 would imply about 5 hour precision.

Given that the Gallery is usually updated at intervals of at most hours to days normally, a value of 2 or 3 is probably optimal.

Must lie in the range 0 to 8 inclusive.

See Also:

Constant Field Values

_AEP_last_longHash

private volatile long _AEP_last_longHash

Hash (longHash) of last-loaded AEP and timestamp, or -1 if no AEP yet loaded. The elements are never null.

Is volatile to allow lockless (read) access.

Updated by _getAllExhibitProperties().

_AEP_nextLoad

private volatile long _AEP_nextLoad

Time before which we should not attempt to reload the AEP. Is volatile to allow lockless (read) access.

Updated by _getAllExhibitProperties().

Initially zero to force first recomputation.

_slow_op_lock

private final java.util.concurrent.locks.ReentrantLock _slow_op_lock

Reentrant lock avoid concurrent slow operations being attempted; never null. Used by both the internal getAEP and getAEID routines.

Rather than block, concurrent attempts may be vetoed with IOException.

MIN_POLL_MS

private static final int MIN_POLL_MS

Minimum time between polls of the filesystem (ms) when not conserving power.

See Also:

Constant Field Values

MIN_POLL_CONSERVING_MS

private static final int MIN_POLL_CONSERVING_MS

Minimum time between polls of the filesystem (ms) when seriously conserving power.

See Also:

Constant Field Values

AEP_LOAD_ESTIMATED_MIN_BYTES_REQUIRED

private static final int AEP_LOAD_ESTIMATED_MIN_BYTES_REQUIRED

Wild guess at some of AEP-load fixed memory overhead, 16MB+ here seems like a good start; strictly positive.

See Also:

Constant Field Values

THUMBNAIL_SUFFIX

public static final java.lang.String THUMBNAIL_SUFFIX

Suffix appended to serialised thumbnail class data for each exhibit. Such files are beside the original exhibit file, are prefixed with FileTools.F_permPrefix, and suffixed with this (which is chosen so as not to look like an exhibit).

The file is a GZIPped, serialised, ExhibitThumbnails class if present.

See Also:

Constant Field Values

THUMBNAILS_ARE_GZIPPED

public static final boolean THUMBNAILS_ARE_GZIPPED

If true, the serialised thumbnail files are compressed (GZIPped) on disc. This might cost a little time but might possibly save a little space. Note that thumbnails may in any case be GZIPped in transit to a slave server.

See Also:

Constant Field Values

_haveWarnedNoTNDir

private volatile boolean _haveWarnedNoTNDir

If true, we have warned about thumbnail directory absence. Starts, false; may be cleared periodically. Set true when we have warned.

Volatile to allow read/write without a lock.

Not critical to correct operation, so we can tolerate races and sloppy handling.

FS_LOG_INTERVAL_MS

private static final int FS_LOG_INTERVAL_MS

Interval between logging filesystem accesses (ms); strictly positive. Should be short enough to reveal operations, possibly unwanted, that 'wake up' the filesystem (eg cause it to power-up, get mounted, etc).

Should be long enough to eliminate most uninteresting stuff such as the multiple accesses to load/examine one file. Keeping the interval to under a minute will most likely catch most 'wake-ups'.

See Also:

Constant Field Values

countFSAccess

private static int countFSAccess

Count of filesystem access operations logged including those not displayed; never null. Accessed under _FSLogLock.

lastLoggedFSAccess

private static long lastLoggedFSAccess

Time that last filesystem access was logged; initially zero. Accessed under _FSLogLock.

_FSLogLock

private static final java.util.concurrent.locks.ReentrantLock _FSLogLock

Lock for logging filesystem access; never null.

GMTCalendar

private static final java.util.Calendar GMTCalendar

Our private static instance of a GMT calendar. No one must change the ZONE or DST offset.

dateFmtEHFile

private static final java.text.SimpleDateFormat dateFmtEHFile

Format we use to insert into event history file name.

timestampFmtEHFile

private static final java.text.SimpleDateFormat timestampFmtEHFile

Format we use for timestamp to insert into event log file name.

MAX_EV_LOG_LINE_LENGTH

private static final int MAX_EV_LOG_LINE_LENGTH

Max log line length for event log.

See Also:

Constant Field Values

_aTWL_lock

private static final java.util.concurrent.locks.Lock _aTWL_lock

Private lock for _appendToEventLogs(), static to serialise all filesystem access.

_VAR_SAVE_INTERVAL_MS

private static final int _VAR_SAVE_INTERVAL_MS

Minimum interval between saves, ms; strictly positive. Intended to avoid losing too much data at shut-down, yet avoid wearing out the disc (or other storage such as Flash) and consuming CPU time with redundant work.

Something from tens of seconds to a few minutes is probably about right.

See Also:

Constant Field Values

_nextVarSaveEarliest

private transient volatile long _nextVarSaveEarliest

Earliest time after which to allow the next event save (zero if no save yet completed). Marked volatile to allow thread-safe lock-free access.

varMgr

private final BasicVarMgr varMgr

Our set of system variables. Persistent values are loaded from disc on first attempt to get and variable(s) if possible.

We expect this to be at the top of the data pipe on the master server, and thus handle all the system non-local variables, plus all the local variables of the master.

If any persistent values are set then they all get written to disc as compressed serialised data on the next poll(). (Doing saves on each poll() helps reduce expensive disc writes by grouping multiple updates into one save if they are happening rapidly.)

The current set or variables (including locals) is held in memory, which also allows us to merge globals if required.

The varMgr is marked as an end-point so that it will generate and return a unique local system ID.

This is assumed not to need to filter out duplicate updates (nor bad timestamps) which is slow and may not even matter here. Any network connection upstream of us should do such filtering.

We have the varMgr screen out repeats and bad timestamps, etc.

eventsToLog

private final java.util.Vector<SimpleVariableValue> eventsToLog

In-order List/Queue thread-safe list of persistent events to be logged; never null. Remove items FIFO or lock the queue and remove all the items in one go.

We don't synchronously save these data.

MAX_LOG_ENTRIES_QUEUED

public static final int MAX_LOG_ENTRIES_QUEUED

Maximum number of log entries to hold in memory before flushing; strictly positive. Should be small enough not to impact memory too much (including the entailed effects of intern()ed/singleton values and so on) but large enough that generally the periodic poll()-driven flush will take care of things, keeping setVariable() and setVariables() fast.

Made public to help other pipeline cacheing stages set related limits.

See Also:

Constant Field Values

ExhibitDataFileSource

public ExhibitDataFileSource(SimpleLoggerIF logger)

Create instance. Loads any persisted system variable values.

Parameters:

logger - logger to write to; null if no logging to be done

getStaticCacheFileName

private static final java.io.File getStaticCacheFileName()

The cached exhibit-data file to read; never null. This is relative to the data directory.

We don't construct the exhibit static data ourselves when called by any of the SimpleExhibitPipelineIF methods, but instead read a GZIPed serialised AllExhibitProperties object at the file given by this value, throwing an IOException if it cannot be read.

The createStaticCacheFile() routine can be used to create this file and do as much precomputation and preparation of thumbnails, etc, as possible off-line. Typically a command-line program would be used to run this each time the exhibits are changed, and the master will simply re-read the serialised file periodically. It is important in this case that the file be replaced atomically. The createStaticCacheFile() tries to create the file if it does not exist.

getStaticHashFileName

private static final java.io.File getStaticHashFileName()

The name of the AEP longHash file; never null. Typically, removing this file, or specifying an old/extant AEP hash different to this, will force the AEP to be reconstructed from the filesystem.

This is relative to the data directory.

Never touches the filesystem; simply constructs the name.

getLastAEPHashIfAny

private static final java.lang.Long getLastAEPHashIfAny()

Get the last/cached AEP longHash file content; null iff not present/readable. The elements are never null.

getStaticAttr

public ExhibitStaticAttr getStaticAttr(Name.ExhibitFull name)
                                throws java.io.IOException

Get the static attributes for a given exhibit; null if no such exhibit.

Specified by:

getStaticAttr in interface SimpleExhibitPipelineIF

Throws:

java.io.IOException - if the operation cannot be completed due to I/O restrictions or failure

_getStaticAttr

private static ExhibitStaticAttr _getStaticAttr(java.lang.String fileRoot,
                                                Name.ExhibitFull name)
                                         throws java.io.IOException

Get the static attributes for a given exhibit; null if the named exhibit does not exist.

This computes a new uncached value on each call.

TODO: prevent this from blocking indefinitely, even with a filesystem hang.

Throws:

java.io.IOException

getRawFile

public void getRawFile(java.nio.ByteBuffer buf,
                       Name.ExhibitFull exhibitName,
                       int position,
                       boolean dontCache)
                throws java.io.IOException

Read a chunk of the raw exhibit binary into the supplied buffer. The start position and an implied maximum read length are supplied. The start must be non-negative and no larger than the exhibit data length.

The call may return less than the the buffer capacity, though will block until it has read at least one byte unless at EOF or for a zero-byte request; this will be clear from the state of the buffer.

If a zero-byte request is made then the file may not actually be accessed.

This goes directly to the filesystem for each call.

TODO: prevent this from blocking indefinitely, even with a filesystem hang.

Specified by:

getRawFile in interface SimpleExhibitPipelineIF

Parameters:

buf - the buffer into which to read the data; must be non-null, in put()able state, and with remaining capacity of at least the requested number of bytes

exhibitName - the full name of the exhibit to read from; never null and must be syntactically valid

position - position/index of first byte in exhibitFile to read; non-negative

dontCache - if true, this is a hint not to attempt to cache this or displace anything from extant caches for this data as it may for example be precaching or random activity; by default callers should leave this false to allow cacheing

Throws:

java.io.IOException - for requests that cannot be fulfilled because of I/O restrictions or problems, such as link failure or an upper bound on the length of a request

getAllExhibitImmutableData

public AllExhibitImmutableData getAllExhibitImmutableData(long oldStamp)
                                                   throws java.io.IOException

Gets all static exhibit data if its timestamp is not that specified. If the time specified is negative then the object will be returned unconditionally.

If no exhibits are currently installed a then default set with a zero timestamp is returned.

If the caller's copy appears to be up-to-date (eg the oldStamp matches that that we would have been returned) then null is returned.

Specified by:

getAllExhibitImmutableData in interface SimpleExhibitPipelineIF

Throws:

java.io.InterruptedIOException - if another expensive call is already in progress; retry later

java.io.IOException - if the operation cannot be completed due to I/O restrictions or failure

_getAllExhibitImmutableData

public static AllExhibitImmutableData _getAllExhibitImmutableData(java.lang.String fileRoot,
                                                                  long oldStamp,
                                                                  boolean careful)
                                                           throws java.io.IOException

Gets all immutable exhibit data if its timestamp is not that specified. If the time specified is negative the object will be returned unconditionally.

This does not use any cache, and computes afresh on each call, and is thread-safe.

If no exhibits are currently installed then a default set with a zero timestamp is returned.

If the caller's copy appears to be up-to-date (eg the oldStamp matches that that we would have been returned) null is returned.

We make the timestamp we use for the whole collection be the most significant bits of the latest-stamped exhibit and the least significant bits of the collection timestamp XORed with a hash over all the (sorted) ExhibitStaticAttr entries. We don't have many least significant bits, so we run a small-ish risk of missing some changes. In fact, we are quite rough-and-ready about the hash too!

We know that many (typically 10) of the least significant bits of the timestamp would not carry information anyway, eg with a one-second granularity in the UNIX filesystem.

To avoid causing too much confusion with our somewhat-faked timestamp, we limit it to between 1 and the current time of day in the worst case, though because we ensure that the fake timestamp calculated is no later than the timestamp of the newest exhibit then unless there is lots of clock-skew between our host and the file server we should not see this latter clamp actually used. The latter limit, if needed, can make the system inefficient just after a new exhibit has been added since our fake timestamp may seem to change on every call.

Parameters:

careful - if true, magic numbers of exhibits are checked and other extra-careful checking is done; this should be the default usage

Throws:

java.io.IOException

getAllExhibitProperties

public AllExhibitProperties getAllExhibitProperties(long oldHash)
                                             throws java.io.IOException

Gets set of all exhibit properties if its hash is not that specified. If the hash specified is negative then the AEP will be loaded and returned unconditionally but this is likely to be expensive; call with the longHash of any extant item if at all possible.

If no exhibits are currently installed then a default empty set with a zero timestamp is returned.

If the caller's copy appears to be up-to-date (ie the oldHash matches that that would have been returned) then null is returned.

Note that a full load from the filesystem is likely to be slow/expensive.

Specified by:

getAllExhibitProperties in interface SimpleExhibitPipelineIF

Throws:

java.io.InterruptedIOException - if another expensive call is already in progress or we cannot otherwise currently attempt to (re)load an AEP; retry later

java.io.IOException - if the operation cannot be completed due to I/O restrictions or failure

_getAllExhibitProperties

private AllExhibitProperties _getAllExhibitProperties(long oldHash,
                                                      boolean careful,
                                                      boolean forceLoadFromFilesystem)
                                               throws java.io.IOException

Gets set of all exhibit properties if its hash is not that specified. If the hash specified is negative then the AEP will be loaded and returned unconditionally but this is likely to be expensive; call with the longHash of any extant item if at all possible.

If no exhibits are currently installed then a default empty set with a zero timestamp is returned.

If the caller's copy appears to be up-to-date (eg the oldHash matches that that would have been returned) then null is returned.

This is more averse to checking/reloading the AEP if conserving power.

Parameters:

careful - if true then magic numbers of exhibits are checked and other extra-careful checking is done; this should be the default usage

forceLoadFromFilesystem - if true, ignore any cached (hash) data and always load data from the filesystem

Throws:

java.io.InterruptedIOException - if another expensive call is already in progress or we cannot otherwise currently attempt to (re)load an AEP; retry later

java.io.IOException

_recomputeAEP

private AllExhibitProperties _recomputeAEP(long oldHash,
                                           boolean careful,
                                           boolean forceLoadFromFilesystem)
                                    throws java.io.IOException,
                                           java.io.InterruptedIOException

Do the slow and memory-intensive construction of the AEP from the filesystem.

Throws:

java.io.IOException

java.io.InterruptedIOException

makeExhibitDataSource

public AllExhibitProperties.ExhibitDataSource makeExhibitDataSource()

Wrap this instance as an ExhibitDataSource. Only really meant for internal use, but may be useful for low-level use of the file-based data.

getGenProps

public GenProps getGenProps(long oldStamp)
                     throws java.io.IOException

Gets the general properties as a GenProps object if its timestamp is not that specified. If the time specified is negative the object will be returned unconditionally.

If no props are currently installed/available a default set with a zero timestamp is returned.

If the caller's copy appears to be up-to-date (eg the oldStamp matches that that we would have been returned) null is returned.

This computes a new uncached value on each call.

Specified by:

getGenProps in interface SimpleExhibitPipelineIF

Throws:

java.io.IOException - if the operation cannot be completed due to I/O restrictions or failure

getGenSecProps

public java.util.Properties getGenSecProps(long oldStamp)
                                    throws java.io.IOException

Gets the generic security properties as a Properties object if its timestamp is not that specified. If the time specified is negative the object will be returned unconditionally.

If no props are currently installed/available a default set with a zero timestamp is returned.

If the caller's copy appears to be up-to-date (eg the oldStamp matches that that would have been returned) null is returned.

This computes a new uncached value on each call.

These generic properties are fetchable over the network, for example, and need not be present locally at each host the JVM runs on.

Specified by:

getGenSecProps in interface SimpleExhibitPipelineIF

Throws:

java.io.IOException - if the operation cannot be completed due to I/O restrictions or failure

turnBackOnMutedWarnings

private void turnBackOnMutedWarnings()

Turn back on warnings we have muted. We do this occasionally so that persistent faults will show up in the log and don't just vanish.

Avoids locking unless really necessary.

logFSAccess

private static void logFSAccess(java.lang.String detail,
                                boolean force)

Log access to data filesystem. Useful for tracing unwanted activity.

May limit output (eg to once every few minutes) to still give traceability of 'wakeup' operations.

Parameters:

force - if true then force printing

getThumbnails

public ExhibitThumbnails getThumbnails(Name.ExhibitFull name,
                                       boolean create)
                                throws java.io.IOException

Gets the thumbnails for an exhibit; null if not (currently) available. In this implementation we never try to create thumbnails (we assume that the source directory is read only for example), but we'll return one if we can find one.

Thumbnails are looked for by default in the exhibits directory tree beside the source exhibits, but a LocalProps value can override this so that source data can be separated from derived data.

Specified by:

getThumbnails in interface SimpleExhibitPipelineIF

Parameters:

create - if true then may try to create and cache missing thumbnails, but creating the thumbnail is only attempted if cacheing is likely to be successful because this should be a once-only operation per exhibit

Throws:

java.io.IOException - if the operation cannot be completed due to I/O restrictions or failure

poll

public void poll(GenProps gp)

Poll periodically. Periodically save the system variables.

Specified by:

poll in interface SimpleExhibitPipelineIF

_saveSystemVariables

private void _saveSystemVariables(BasicVarMgr vars,
                                  boolean force)

Save the system variables (and logs). Mainly save the histories of persistent events.

We save these with a history so that in effect we have backups if something bad happens (though this implies that some other housekeeping will have to clear up excess copies, etc, in order to avoid space requirements growing without bound).

The data is saved in the persistent data area.

Reports but does not propagate exceptions.

Parameters:

vars - set of variables to save

force - if true, force an immediate complete save

_appendToEventLogs

private void _appendToEventLogs()
                         throws java.io.IOException

Write out the accumulated event values to disc. These are written to date-stamped files to allow automatic "rolling".

We keep a separate file for each named event, and append events to each appropriate file in order with a record of the form:

     YYYYMMDD-HHmmss.SSS eventName eventValue
 

with a null value being indicated by the value "null".

Note that the timestamp is that of the value, not the time that the event is received or written.

We may cache file handles to reduce the cost of opening and closing files for each event, but in any case will have closed all log files opened by us by the time this routine returns.

We synchronise on a private static lock to try to prevent two instances of this running concurrently and leading to file corruption or misordering the logs; a second concurrent attempt is silently vetoed.

Throws:

java.io.IOException

_getEventHistoryStorageDir

private static java.io.File _getEventHistoryStorageDir()

Get directory used for storing system-variable persistent event histories; never null. Though never null, the indicated directory may not exist or may otherwise not be usable.

We derive this from the LocalProps value.

_getEventLogDir

private static java.io.File _getEventLogDir()

Get directory used for storing system-variable persistent event logs; never null. Though never null, the indicated directory may not exist or may otherwise not be usable.

We get derive this from the LocalProps value.

setVariable

public void setVariable(SimpleVariableValue newValue)
                 throws java.io.IOException

Set variable value; persistent values will eventually go to disc.

Specified by:

setVariable in interface BasicVarMgrInterface

Parameters:

newValue -

Throws:

java.io.IOException

queueNewEventToLog

private void queueNewEventToLog(SimpleVariableValue newValue)
                         throws java.io.IOException

Queue one new event to be logged. If the number of accrued events becomes too large then flush to the filesystem synchronously and liberate some memory. (Generally we hope that background polling will flush logs as needed.)

Throws:

java.io.IOException

setVariables

public int setVariables(SimpleVariableValue[] newValues)
                 throws java.io.IOException

Set variable values; persistent values will eventually go to disc.

Specified by:

setVariables in interface BasicVarMgrInterface

Parameters:

newValues -

Returns:

the number of variable values set (the length of the array); never negative, never more than the number passed in

Throws:

java.io.IOException

getVariable

public SimpleVariableValue getVariable(SimpleVariableDefinition var)

Get variable value; persistent values may have come from disc.

Specified by:

getVariable in interface BasicVarMgrInterface

Parameters:

var - definition of variable to fetch; never null

getVariables

public SimpleVariableValue[] getVariables(long changedSince)

Get variable values; persistent values may have come from disc.

Specified by:

getVariables in interface BasicVarMgrInterface

syncVariables

public void syncVariables(boolean force)

Synchronise variable values. This implementation does not force a flush to disc.

Specified by:

syncVariables in interface SimpleVariablePipelineIF

Parameters:

force - if true, this will force a full write flush, a full sync upstream, then full read with getVariables(-1), to get the effect of a full "barrier"; otherwise, in general, a more incremental and non-propagating mode is used which still does a write flush but may chose to do a partial read of "new" upstream values

getEventValue

public EventVariableValue getEventValue(SimpleVariableDefinition def,
                                        EventPeriod intervalSelector,
                                        boolean current)

Get the current partial, or previous full, event set at the specified interval; never null. This is a simplified interface to return either the current event set that is being collected, or the previous completed set.

The current set is the most timely, but may not contain enough data to be meaningful if the new interval has just started.

The previous set is complete and thus most likely to have enough samples to be useful, but is not completely current.

If a "previous" value returned from our local store is not marked as authoritative, then we convert it to an authoritative value, write that back to our local store and return it. (The "current" value is generally not authoritative.)

Specified by:

getEventValue in interface BasicVarMgrInterface

Parameters:

def - event definition (must be for an event); never null

intervalSelector - one of EVENT_INTERVAL_SELECTOR_xxx values

current - if true the current event set is returned, else the previous complete set is returned

Returns:

requested event set; may be empty but never null if requested set not available

getEventValues

public EventVariableValue[] getEventValues(SimpleVariableDefinition def,
                                           EventPeriod intervalSelector,
                                           long intervalNumber,
                                           java.util.BitSet whichValues)

Get the specified event sets for the specified intervals; never null. This allows retrieval of zero or more event sets for the specified interval size.

Requests for more than SystemVariables.EVENT_SAMPLES_RETAINED in the past (or for the future!) cannot be satisfied and data will not be returned for them.

Usually not more than SystemVariables.EVENT_SAMPLES_RETAINED samples will be returned in response to any one request as a safety measure.

(An implementation that is not an end-point may go upstream to fetch missing values and cache them to satisfy future requests.)

If a "previous" value (older than the current interval) returned from our local store is null or not marked as authoritative, then we convert it to an authoritative value, write that back to our local store and return it. (The "current" value is generally not authoritative, and entries in the future cannot be so.)

Specified by:

getEventValues in interface BasicVarMgrInterface

Parameters:

def - event definition (must be for an event); never null

intervalSelector - one of EVENT_INTERVAL_SELECTOR_xxx values

intervalNumber - a time (as from System.currentTimeMillis()) which identifies the first interval for which data is potentially required; if too far in the past or future then possibly no data will be available, zero is used to access the "all" bucket

whichValues - each true bit represents a slot for which data is required, bit 0 indicating data from the slot within which firstIntervalTime is located, bit 1 the previous slot, etc

Returns:

as many of the requested values as available, at least long enough to return all the available values, with [0] corresponding to bit 0 in the BitSet; may contain nulls or be zero-length but is never null

thumbnailRelPath

private static java.lang.String thumbnailRelPath(Name.ExhibitFull exhibitName)

Creates the name of a thumbnail starting at the thumbnail base directory from a full exhibit name.

createStaticCacheFile

private static AllExhibitProperties createStaticCacheFile(SimpleLoggerIF logger,
                                                          java.io.File outputFile,
                                                          boolean quick,
                                                          boolean recompute,
                                                          boolean fixaccessions)
                                                   throws java.io.IOException

Computes an AllExhibitProperties object and saves it to outputFile. The object is saved serialised and GZIPed.

The save is avoided if we are not forcing a complete recompute and the exhibit hash has not changed, ie we try to avoid unnecessarily churning the filesystem and/or the master server.

It almost certainly does not make sense for the WAR_SYSPROPNAME_WARONLY_STATICCACHEFILE property to be set; you could get a similar result by copying the file.

In passing, this may update ancillary state such as computable properties and thumbnails.

Parameters:

logger - TODO

outputFile - desired location of cache file; not null

quick - if true, attempt to be as quick as possible, else if false, magic numbers of exhibits are checked and other extra-careful checking is done; this should be the default usage

recompute - if true, recompute all derived data from scratch, eg do not reload any extant cache file

Returns:

AllExhibitProperties (non-null)

Throws:

java.io.IOException

createAccessionFiles

private static boolean createAccessionFiles(AllExhibitImmutableData aeid)
                                     throws java.io.IOException

Create accessions files.

Returns:

true if any accessions files were created or fixed

Throws:

java.io.IOException

getProperties

public java.util.Properties getProperties(SimpleExhibitPipelineIF.PropsKey key,
                                          long versionID)
                                   throws java.io.IOException

Get requested Properties selected by key and versionID. Fetches a Properties set unconditionally (versionID == -1) else if the versionID presented is not current.

Specified by:

getProperties in interface SimpleExhibitPipelineIF

Parameters:

key - selector (with possible embedded sub-key) for desired properties set; never null

versionID - if -1 then map is always returned if available, else must be non-negative and null is returned if the versionID presented matches that of the current version (ie if the caller has presumably got the up-to-date version); may be a timestamp or a hash or other value, and by convention is zero only for an empty properties set

Returns:

null, or Properties map guaranteed to contain only String keys and values

Throws:

java.io.IOException

main

public static void main(java.lang.String[] args)

Run from the command-line with a single argument (the output file name). There is an optional -quick first argument, which if present which attempts to cap the time spent in one run (typically to no more than a few minutes) so that any work to be done can be done incrementally in several passes if need be.

When making a new static cache file we always try to reload the old one (unless the -recompute flag is set, in which case we will not load it) to save lots of time (especially for the computed properties) and possibly preserve the hashNotChangedSince value.

If the -fixaccessions flag is passed then we try to fix existing accessions files that are missing information, eg new checksum types, though we will never try to "fix" an incorrect checksum for example (that will always result in a warning on the console.)

If the -checkexhibits flag is true then we try to verify that each exhibit's exhibit length, timestamp and hash matches the values captured in the associated accession file. We may check additional data (eg FEC data) in future. Note that this option overrides any other options, and avoids writing anything to disc.

checkExhibitData

private static void checkExhibitData(boolean quick,
                                     java.io.File dataDir)
                              throws java.io.IOException

Check exhibit data for corruption.

Parameters:

quick - take some optional speed-ups

Throws:

java.io.IOException - in case of difficulty

getStratum

public Stratum getStratum()

Assumes that this instance is the root/master and so returns Stratum.ROOT; non-null.

Specified by:

getStratum in interface SimpleExhibitPipelineIF

destroy

public void destroy()

Shut down the data pipeline. Saves any pending state (eg log entries and variables) to disc if possible.

Has no upstream components to shut down nor significant resources (memory) to release.

Specified by:

destroy in interface SimpleExhibitPipelineIF