The library library(semweb/rdf_litindex.pl) exploits the
primitives of section 4.5.1 and the
NLP package to provide indexing on words inside literal constants. It
also allows for fuzzy matching using stemming and‘sounds-like'
based on the double metaphone algorithm of the NLP package.
- rdf_find_literals(+Spec,
-ListOfLiterals)
- Find literals (without type or language specification) that satisfy
Spec. The required indices are created as needed and kept
up-to-date using hooks registered with rdf_monitor/2.
Numerical indexing is currently limited to integers in the range ±2^30
(±2^62 on 64-bit platforms). Spec is defined
as:
- and(Spec1, Spec2)
- Intersection of both specifications.
- or(Spec1, Spec2)
- Union of both specifications.
- not(Spec)
- Negation of Spec. After translation of the full specification
to
Disjunctive Normal Form (DNF), negations are only allowed
inside a conjunction with at least one positive literal.
- case(Word)
- Matches all literals containing the word Word, doing the
match case insensitive and after removing diacritics.
- stem(Like)
- Matches all literals containing at least one word that has the same stem
as Like using the Porter stem algorithm. See NLP package for
details.
- sounds(Like)
- Matches all literals containing at least one word that‘sounds
like'
Like using the double metaphone algorithm. See NLP package
for details.
- prefix(Prefix)
- Matches all literals containing at least one word that starts with
Prefix, discarding diacritics and case.
- between(Low, High)
- Matches all literals containing an integer token in the range
Low..High, including the boundaries.
- ge(Low)
- Matches all literals containing an integer token with value
Low or higher.
- le(High)
- Matches all literals containing an integer token with value
High or lower.
- Token
- Matches all literals containing the given token. See tokenize_atom/2
of the NLP package for details.
- rdf_token_expansions(+Spec,
-Expansions)
- Uses the same database as rdf_find_literals/2
to find possible expansions of Spec, i.e. which words‘sound
like',‘have prefix', etc. Spec is a compound expression
as in rdf_find_literals/2.
Expansions is unified to a list of terms
sounds(Like,
Words), stem(Like, Words) or prefix(Prefix,
Words). On compound expressions, only combinations that provide
literals are returned. Below is an example after loading the ULAN2Unified
List of Artist Names from the Getty Foundation. database
and showing all words that sounds like‘rembrandt' and appear
together in a literal with the word‘Rijn'. Finding this result
from the 228,710 literals contained in ULAN requires 0.54 milliseconds
(AMD 1600+).
?- rdf_token_expansions(and('Rijn', sounds(rembrandt)), L).
L = [sounds(rembrandt, ['Rambrandt', 'Reimbrant', 'Rembradt',
'Rembrand', 'Rembrandt', 'Rembrandtsz',
'Rembrant', 'Rembrants', 'Rijmbrand'])]
Here is another example, illustrating handling of diacritics:
?- rdf_token_expansions(case(cafe), L).
L = [case(cafe, [cafe, caf\'e])]
- rdf_tokenize_literal(+Literal,
-Tokens)
- Tokenize a literal, returning a list of atoms and integers in the range
-1073741824 ... 1073741823. As tokenization is in general
domain and task-dependent this predicate first calls the hook
rdf_litindex:tokenization(Literal, -Tokens). On failure it
calls tokenize_atom/2
from the NLP package and deletes the following: atoms of length 1,
floats, integers that are out of range and the english words and, an, or, of,
on, in, this and the.
Deletion first calls the hook rdf_litindex:exclude_from_index(token,
X). This hook is called as follows:
no_index_token(X) :-
exclude_from_index(token, X), !.
no_index_token(X) :-
...
‘Literal maps' provide a relation between literal values,
intended to create additional indexes on literals. The current
implementation can only deal with integers and atoms (string literals).
A literal map maintains an ordered set of keys. The ordering
uses the same rules as described in section
4.5. Each key is associated with an ordered set of values.
Literal map objects can be shared between threads, using a locking
strategy that allows for multiple concurrent readers.
Typically, this module is used together with rdf_monitor/2
on the channals new_literal and old_literal to
maintain an index of words that appear in a literal. Further abstraction
using Porter stemming or Metaphone can be used to create additional
search indices. These can map either directly to the literal values, or
indirectly to the plain word-map. The SWI-Prolog NLP package provides
complimentary building blocks, such as a tokenizer, Porter stem and
Double Metaphone.
- rdf_new_literal_map(-Map)
- Create a new literal map, returning an opaque handle.
- rdf_destroy_literal_map(+Map)
- Destroy a literal map. After this call, further use of the Map
handle is illegal. Additional synchronisation is needed if maps that are
shared between threads are destroyed to guarantee the handle is no
longer used. In some scenarios rdf_reset_literal_map/1
provides a safe alternative.
- rdf_reset_literal_map(+Map)
- Delete all content from the literal map.
- rdf_insert_literal_map(+Map,
+Key, +Value)
- Add a relation between Key and Value to the map.
If this relation already exists no action is performed.
- rdf_insert_literal_map(+Map,
+Key, +Value, -KeyCount)
- As rdf_insert_literal_map/3.
In addition, if Key is a new key in
Map, unify KeyCount with the number of keys in Map.
This serves two purposes. Derived maps, such as the stem and metaphone
maps need to know about new keys and it avoids additional foreign calls
for doing the progress in
rdf_litindex.pl.
- rdf_delete_literal_map(+Map,
+Key)
- Delete Key and all associated values from the map. Succeeds
always.
- rdf_delete_literal_map(+Map,
+Key, +Value)
- Delete the association between Key and Value from
the map. Succeeds always.
- [det]rdf_find_literal_map(+Map,
+KeyList, -ValueList)
- Unify ValueList with an ordered set of values associated to
all keys from KeyList. Each key in KeyList is
either an atom, an integer or a term
not(Key). If not-terms
are provided, there must be at least one positive keywords. The
negations are tested after establishing the positive matches.
- rdf_keys_in_literal_map(+Map,
+Spec, -Answer)
- Realises various queries on the key-set:
- all
- Unify Answer with an ordered list of all keys.
- key(+Key)
- Succeeds if Key is a key in the map and unify Answer
with the number of values associated with the key. This provides a fast
test of existence without fetching the possibly large associated value
set as with rdf_find_literal_map/3.
- prefix(+Prefix)
- Unify Answer with an ordered set of all keys that have the
given prefix. Prefix must be an atom. This call is intended
for auto-completion in user interfaces.
- ge(+Min)
- Unify Answer with all keys that are larger or equal to the
integer Min.
- le(+Max)
- Unify Answer with all keys that are smaller or equal to the
integer Max.
- between(+Min, +Max)
- Unify Answer with all keys between Min and Max
(including).
- rdf_statistics_literal_map(+Map,
+Key(-Arg...))
- Query some statistics of the map. Provides keys are:
- size(-Keys, -Relations)
- Unify Keys with the total key-count of the index and
Relation with the total Key-Value
count.