Add appropriate tools for special startup behavior
There are a couple of cases where it's desirable with special startup behavior when starting the primary log server (and maybe for secondary as well).
- Starting a new log, currently requires running the sigsum-mktree to create an empty sth (signed tree head) file.
- Starting primary after failover from secondary: Needs changes to the trillian tree state, done like this in the integration test script: https://git.glasklar.is/sigsum/core/log-go/-/blob/main/integration/test.sh#L172, using trillian's update tree command. This is not particularly user friendly. May also need opposite transition, if a secondary was promoted to primary temporarily, and switched back to secondary once a new primary has been deployed.
- Also after failover, it's desirable that the new primary (previously secondary) creates a new signed tree head based on its latest local tree head (due to state logic in the primary server, this is guaranteed to not be behind the tree advertised by the previous primary before it failed). Currently not properly implemented, one would need to copy the sth file from the primary, which isn't possible in case of catastrophic failure.
We need some utilities to make this easier for users, either as separate command line tools or integrated into the log binaries.
For (2) in particular, it's desirably for the server to check trillian's tree state (LOG, suitable for primary, or PREORDERED_LOG for a secondary) at startup. If state is unexpected, possible action would be to either exit with a failure, exit with an error message quoting the updatetree commands (including appropriate --admin_server and --tree_id options) to change the state, or automatically switch state.
Integrating in the log server, rather than separate commands like sigsum-mktree, has the advantage that there's less risk that commands are run with different and inconsistent config than the actual server. On the other hand, it makes it tempting to have startscripts/ansible always enable, e.g., automatic creation of sth file, which could mask problems.
One alternative would be to enable magic contents of the sth file, e.g., a single line "startup=" instead of the usual contents. We could have "startup=empty", "startup=transition-secondary-to-primary" etc to specify action at startup. Once that action is done, the file would be deleted or replaced with a normal sth file. This would ensure that special action is done only when explicitly requested, and not by routine.