2013-02-15 Martyn Russell Release 0.15.2 Merge branch 'fts4-fixes' 2013-02-14 Martyn Russell ontology: Add a comment that tracker:fulltextNoLimit is now deprecated Could not mark this as deprecated because it's not a class libtracker-fts: Fix unit test ontologies, removed tracker:fulltextNoLimit functional-tests: Removed tracker:fulltextNoLimit libtracker-data: Removed tracker:fulltextNoLimit tracker-sparql: Fixed documentation, removed tracker:fulltextNoLimit ontology: Remove use of tracker:fulltextNoLimit now it's deprecated 2013-02-14 Carlos Garnacho tests: tweak FTS tests to cope with the word length changes This has caused some tests to return more results than before, so add these extra results to the expected output, as the tests still prove their point. fts: Remove min word length limit from configuration It isn't used anymore, so remove the dangling setting. 2013-02-11 Martyn Russell Merge branch 'fts4' 2013-02-08 Carlos Garnacho Bump database version This is needed after the fts4 changes 2013-02-06 Martyn Russell libtracker-miner: Fixed GNOME password provider using libsecret - "user" --> "username" for the attributes lookup - Was checking for SecretItem != NULL to error up the stack Thanks to Stef Walter for the help here. 2013-02-05 Martyn Russell tracker-info: Fixed memory leak where urn wasn't freed. tracker-info: Fixed memory leak when using prefixes for shorthand properties e.g. nie:url instead of fully qualified URLs tracker-sparql: Support fts:offsets(?urn) with shorthand properties e.g. nie:title:0 nid3:contentType:24 nid3:contentType:43 2013-02-04 Martyn Russell libtracker-data: Use g_hash_table_unref() not _destroy() libtracker-data: Use a g_message() for cases where FTS is disabled 2013-02-04 Carlos Garnacho tracker-needle: display snippet where available Handle FTS queries more generically FTS functions and other SQL functions like COUNT() don't mix well, so handle it more generically by joining with the FTS table at the outmost level, so those functions are handled separately. Use fts4 external content tables on Tracker FTS FTS support on Tracker is now implemented using external content support available in sqlite >= 3.7.9 FTS4. FTS tables created this way fetch data from a single table/view, for this purpose an intermediate view has been created to interface with the FTS table. As this view is mainly queried by ID (both when populating the FTS contents, and when querying throught the FTS table), queries are fast enough on it. As FTS indirectly points to the data on the real tables where tracker data is stored, strings themselves are stored only once in the database, so there is no impact in database size when compared to the previous custom FTS code. Performance had little changes too from testing. Detect at configure time whether FTS needs compiling Recent sqlites (>3.7.9) with FTS support compiled have all what Tracker requires to handle FTS, so only compile it optionally if no FTS support was detected in the sqlite library. Update FTS code libtracker-data: Fix FTS initialization Use a define that actually exists around tracker_fts_init(), which also checked for the inverted value. libtracker-data: Fix fts detection when constructing SQL It would previously miss the FTS binding, depending on the order of the clauses. so something like { ?u fts:match 'foo'; a nie:DataObject } would return results whereas { ?u a nie:DataObject ; fts:match 'foo' } didn't. libtracker-fts: Fix reentrancy issues in the TrackerTokenizer The internal TrackerParser must be per-cursor, as several cursors may be opened simultaneously. libtracker-data: Implement fts:snippet() this function takes up to 3 optional parameters after the object, the first 2 parameters are the starting/ending text for the match (defaults to ), and the third one modifies the ellipsis text (defaults to ...) libtracker-data: Perform FTS matching in subquery The outer query will call the functions that require a full row scan only with the elements returned by the inner query, so it would perform better with OFFSET and LIMIT fts: Respect max-words configuration setting libtracker-data: Gather FTS changes to perform a single insert/update per ID libtracker-data: remove useless FTS functions commit/rollback should happen now with the rest of the operations, and update_init() isn't necessary anymore. libtracker-data: Cope with ontology changes in the FTS table If a new property is with tracker:fulltextIndexed, we need to add a new column for it in the FTS table. libtracker-fts: Add a way to modify FTS table columns As ALTER TABLE on virtual tables is only able to do renamings, resort to creating a temporary table, and dump the old values into the new table, which is then renamed. libtracker-fts: Add tracker_offsets() sqlite function This function takes the offsets() output and the FTS property names in order to produce the output as expected by Tracker. the fts:offsets() sparql function has been adapted to use it underneath libtracker-fts: Add the tracker_rank() sqlite function This is now used in FTS searches to provide the rank based on the matched columns and their respective property weight. libtracker-data: Create several columns for FTS data there is now one column per ontology property with FTS enabled, the column name is that of the property. libtracker-data: Add tracker_db_interface_sqlite_fts_delete_text() This function removes from the FTS table the text from a resourceid/propid pair. libtracker-fts: Implement tokenizer FTS module using TrackerParser libtracker-fts: Update to FTS4 The code isn't yet feature complete, but cleans up the tracker integration with the FTS module, which means that fts3* files are taken verbatim from sqlite, and are licenced as such. 2013-02-03 Jens Georg tracker-extract: Port to new gupnp-dlna 2.0 API GUPnP-DLNA 0.9/0.10 changed a lot internally and thus does not deliver a GstDiscovererInfo anymore but instead has a new API that allows guessing the profile from an existing GstDiscovererInfo, so most of the additional logic can be dropped. Fixes GB#691847, tracker-extract: Port to new gupnp-dlna API 2013-02-02 Martyn Russell tracker-extract-playlist: Support nie:title for the playlist Fixed GB#692409, New: playlist extractor should index playlist title if available 2013-02-01 Martyn Russell tracker-needle: Find documents (e.g. PDFs) when searching for creators/publishers Fixes GB#692603, Search can not find pdf files using author names