storage.pl -- Store files on behalve of web clients

The file store needs to deal with versioning and meta-data. This is achieved using gitty.pl, a git-like content-base store that lacks git's notion of a tree. I.e., all files are considered individual and have their own version.

 web_storage(+Request) is det[private]

Restfull HTTP handler to store data on behalf of the client in a hard-to-guess location. Returns a JSON object that provides the URL for the data and the plain file name. Understands the HTTP methods GET, POST, PUT and DELETE.

 update_error(+Error, +Storage, +Data, +File, +URL)[private]

If error signals an edit conflict, prepare an HTTP 409 Conflict page

 follow(+Commit, +SaveDict) is det[private]

Broadcast follow(DocID, ProfileID, [update,chat]) if the user wishes to follow the file associated with Commit.

 request_file(+Request, +GittyDir, -File) is det[private]

Extract the gitty file referenced from the HTTP Request.

Errors

- HTTP 404 exception

 meta_data(+Dict, -Meta, +Options) is det[private]

 meta_data(+Store, +Dict, -PrevMeta, -Meta, +Options) is det[private]

Gather meta-data from the Request (user, peer, identity) and provided meta-data. Illegal and unknown values are ignored.

The meta_data/5 version is used to add information about a fork.

Arguments:

Dict	- represents the JSON document posted and contains the content (data) and meta data (meta).

 storage_get(+Request, +Format, +Options) is det[private]

HTTP handler that returns information a given gitty file.

Arguments:

Format

- is one of swish Serve file embedded in a SWISH application raw Serve the raw file json Return a JSON object with the keys data and meta history(Depth, IncludeHASH) Return a JSON description with the change log diff(RelTo) Reply with diff relative to RelTo. Default is the previous commit.

 gitty_data_or_default(+Dir, +Type, +FileOrHash, -Code, -Meta)[private]

Read a file from the gitty store. I the file is not present, a default may be provided gitty/File in the config directory.

 chat_count(+Meta, -ChatCount) is det[private]

True when ChatCount is the number of chat messages available about Meta.

 random_filename(-Name) is det[private]

Return a random file name from plain nice ASCII characters.

 swish_show(+Options, +Request)[private]

Hande a document. First calls the hook open_hook/2 to rewrite the document. This is used for e.g., permahashes.

 storage_file(?File) is nondet

 storage_file_extension(?File, ?Extension) is nondet

 storage_file_extension_head(?File, ?Extension, -Head) is nondet

 storage_file(+File, -Data, -Meta) is semidet

 storage_meta_data(+File, -Meta) is semidet

True if File is known in the store.

Arguments:

Data	- is a string holding the content of the file
Meta	- is a dict holding the meta data about the file.

 storage_commit(+Hash, -Meta) is semidet

Load the commit data for Hash. This version does not tell us whether Hash is the HEAD or not.

 storage_meta_property(+Meta, -Property)

True when Meta has Property. Defined properties are:

peer(Atom)

Peer address that last saved the file -

 storage_store_term(+Term, -Hash) is det

 storage_load_term(+Hash, -Term) is det

Add/retrieve terms from the gitty store. This is used to create permanent links to arbitrary objects.

 use_gitty_file(+File) is det

 use_gitty_file(+File, +Options) is det

Load a file from the Gitty store. Options are passed to load_files/2. Additional options are:

watch(+Boolean)

If true (default), reload the file if the user saves it.

 collect_messages_as_json(+Goal, -Messages)[private]

Run Goal, collecting messages as produced by print_message/2 in Messages as JSON terms.

 storage_fsck

Enumerate and check the consistency of the entire store.

 storage_repack is det

 storage_repack(+Options) is det

Repack the storage directory. Currently only supports the files driver. For database drivers this is supposed to be handled by the database.

 storage_unpack

Unpack all packed objects of the store. Currently only supports the files driver. For database drivers this is supposed to be handled by the database.

 swish_search:typeahead(+Set, +Query, -Match, +Options) is nondet[multifile]

Find files using typeahead from the SWISH search box. This version defines the following sets:

• file: Search the store for matching file names, matching tag or title.
• store_content: Search the content of the store for matching lines.

To be done

- caching?

- We should only demand public on public servers.

 source_list(+Request)[private]

List source files. Request parameters:

q(Query)

Query is a string for which the following sub strings are treated special:

"..."

A quoted string is taken as a string search $ /.../[xim]* Regular expression search

tag:Tag

Must have tag containing

type:Type

Limit to one of pl, swinb or lnk

user:User

Must have user containing. If User is me must be owned by current user

name:Name

Must have name containing

o(Order)

Order by time (default), name, author or type

offset(+Offset)

limit(+Limit)

display_name

avatar

Weak identity parameters used to identify own documents that are also weakly identified.

Reply is a JSON object containing count (total matches), cpu (CPU time) and matches (list of matching sources)

To be done

- Search the content when searching a .lnk file?

- Speedup expensive searches. Cache? Use external DB?

 search_sources(+Query, -Results) is det[private]

Search the available sources. Query is a dict holding

query.query_string

The original query string.

Query.query

Parsed query string. This is a list of Tag(Value), word(Word), regex(String, Flags) or string(String) (quoted search). The Value is either a string or regex(String, Flags).

Query.auth

Authentication information for the current session

Query.order_by

Field to order on

Query.order

Ordering (one of desc (@>=) or asc (@=<))

Query.limit

Number of results to return

Query.offset

Number of results to skip

Result is a dict holding

• Result.matches The actual hits. The matches are dicts holding these keys

  Match.time

  Time stamp (seconds since 1970)

  Match.tags

  Tags of the document

  Match.author

  Author of the document

  Match.avatar

  Known avatar

  Match.name

  File name

• Result.total Total number of answers
• cpu:CPU time.

This predicate can be hooked using search_sources_hook/2 that uses the same signature. If the hook fails, naive search is performed. The naive algorithm is fine for local installations with a couple of hundreds of files. Public installations need to hook this predicate using a proper full text database.

 visible(+FileMeta, +Auth, +MetaConstraints) is semidet[private]

 owns(+Auth, +Meta, ?How) is semidet[private]

True if the file represented by Meta is owned by the user identified as Auth. If this is a strong identity we must give a strong answer.

To be done

- Weaker identity on the basis of author, avatar properties and/or IP properties.

 matches_meta(+Source, +Auth, +Query) is semidet[private]

True when Source matches the meta-data requirements

 type_constraint(+Query0, -Query, -Type) is det[private]

Extract the type constraints from the query as we can handle that efficiently.

 parse_query(+String, -Query) is det[private]

Parse a query, resulting in a list of Name(Value) pairs. Name is one of tag, user, type, string or regex. Value is one of a string, string(String) (quoted), regex(String, Flags) or word(String).

 source_modified(+Request)[private]

Reply with the last modification time of the source repo. If there is no modification we use the time the server was started.

This is a poor men's solution to keep the client cache consistent. Need to think about a better way to cache searches client and/or server side.

Re-exported predicates

The following predicates are exported from this file while their implementation is defined in imported modules or non-module files loaded by this module.

 storage_file(?File) is nondet

 storage_file_extension(?File, ?Extension) is nondet

 storage_file_extension_head(?File, ?Extension, -Head) is nondet

 storage_file(+File, -Data, -Meta) is semidet

 storage_meta_data(+File, -Meta) is semidet

True if File is known in the store.

Arguments:

Data	- is a string holding the content of the file
Meta	- is a dict holding the meta data about the file.

 storage_file(?File) is nondet

 storage_file_extension(?File, ?Extension) is nondet

 storage_file_extension_head(?File, ?Extension, -Head) is nondet

 storage_file(+File, -Data, -Meta) is semidet

 storage_meta_data(+File, -Meta) is semidet

True if File is known in the store.

Arguments:

Data	- is a string holding the content of the file
Meta	- is a dict holding the meta data about the file.

 storage_file(?File) is nondet

 storage_file_extension(?File, ?Extension) is nondet

 storage_file_extension_head(?File, ?Extension, -Head) is nondet

 storage_file(+File, -Data, -Meta) is semidet

 storage_meta_data(+File, -Meta) is semidet

True if File is known in the store.

Arguments:

Data	- is a string holding the content of the file
Meta	- is a dict holding the meta data about the file.

 storage_file(?File) is nondet

 storage_file_extension(?File, ?Extension) is nondet

 storage_file_extension_head(?File, ?Extension, -Head) is nondet

 storage_file(+File, -Data, -Meta) is semidet

 storage_meta_data(+File, -Meta) is semidet

True if File is known in the store.

Arguments:

Data	- is a string holding the content of the file
Meta	- is a dict holding the meta data about the file.

 use_gitty_file(+File) is det

 use_gitty_file(+File, +Options) is det

Load a file from the Gitty store. Options are passed to load_files/2. Additional options are:

watch(+Boolean)

If true (default), reload the file if the user saves it.

 storage_store_term(+Term, -Hash) is det

 storage_load_term(+Hash, -Term) is det

Add/retrieve terms from the gitty store. This is used to create permanent links to arbitrary objects.

 storage_repack is det

 storage_repack(+Options) is det

Repack the storage directory. Currently only supports the files driver. For database drivers this is supposed to be handled by the database.