libera/#maemo-leste/ Wednesday, 2023-12-06

freemangordonWizzup: that query results in valid id when called from within miner07:44
freemangordonstarting to look like race condition07:44
sunshavia count before the select?10:34
Wizzupfreemangordon: what is the id?11:20
freemangordonurn:uuid:bb8cbccd-320e-456c-9383-9a23124e432811:46
freemangordonbut there is no such id when I select from the shell11:47
Wizzupmaybe it's some id that is itself temporaraily adds11:50
WizzupI wonder if you can just check if there is a result, and see if the first result is "None"11:50
freemangordonor sqlite caches data11:50
freemangordonno, first result is urn\11:51
WizzupI wouldn't blame sqlite3 here, but maybe how tracker uses sqlite311:51
Wizzupaha11:51
Wizzupgot the full query for me?11:51
freemangordon'SELECT ?u {   GRAPH <urn:uuid:472ed0cc-40ff-4e37-9c0c-062d78656540> {    ?u a nfo:FileDataObject ;       tracker:available true ;        a ?class .     FILTER (?class IN (nfo:Media,nfo:Icon,nfo:Document,nmm:Playlist,nmm:MusicPiece,nfo:Audio,nfo:PlainTextDocument,nmm:Video,nfo:SourceCode,nfo:HtmlDocument,nfo:Video,nfo:SoftwareApplication,nfo:EBook,nmm:Photo,nfo:VectorImage,nfo:TextDocument,nfo:PaginatedTextDocument,nfo:FilesystemImage,nfo:Image,nfo:M11:51
freemangordondo you want me to pastebin it?11:52
WizzupI can work with this I think11:52
Wizzupso you say from cli this returns nothing?11:52
Wizzupoh actually it is cut off11:52
freemangordonthis returns no result in the shell and uuid in tracker-miner-fs11:52
freemangordonyes11:52
freemangordonand there is no such uuid in the database11:53
Wizzupis there anything other than uuid in the row?11:53
freemangordonno, see the query11:53
freemangordon?u a nfo:FileDataObject11:53
Wizzupso one possibility is that this is some tmp file11:53
WizzupI've seen tracker-miner-fs find various tmp files in /var/tmp11:53
Wizzupno, hang on, that is not right11:53
WizzupI've seen it on dbus-monitor as picked up by some inotify daemon, but not tracker-miner-fs11:54
Wizzupbut maybe it is some tmp file that it picks up when the program starts up11:54
freemangordonWizzup: i debugged to the pont where sqlite returns result11:55
freemangordon*point11:55
freemangordonthis is *not* tracker caching something or picking up wrong file or whatever11:55
freemangordonit id DB returning stalled data11:56
freemangordontracker sparql -q 'SELECT * WHERE { ?u a nfo:FileDataObject. FILTER (?u = urn:uuid:bb8cbccd-320e-456c-9383-9a23124e4328)}' returns nothing in the console11:56
bencohsqlite showing stalled data might mean that two processes opened the db11:56
Wizzupfreemangordon: you mean that you stopped gdb?11:57
Wizzupstopped it from gdb?11:57
freemangordonbencoh: yes, looks like that11:57
Wizzupsqlite3 has a notion of transations, journals, write ahead log, and so oh, so this doesn't have to be a sqlite3 problem11:57
freemangordonWizzup: stooped? no, stepped :)11:57
freemangordonand yes, I started tracker-miner-fs from gdb11:58
Wizzuplike if tracker-miner-fs starts a transaction, crawls fs, adds something, then runs that query before commit, only it will see it11:58
WizzupI mean you know all of these things, but just saying11:58
freemangordonlook at the select11:58
Wizzupright11:58
freemangordonit has a condition11:58
freemangordonand (i suspect) that gets set by tracker-extract11:58
Wizzuptracker-extract doesn't run at the time of this query though11:59
freemangordonyes11:59
freemangordonand DB as such is correct11:59
freemangordonbut it seems there is cached data in the miner11:59
freemangordonas bencoh said, because we have 2 processes opening the DB11:59
freemangordonI see the same issue with mafw-tracker-source11:59
Wizzupbut only one active process :)12:00
freemangordonwhere sparql queries return some obsolete data12:00
freemangordondoes not matter12:00
Wizzupok, well, this still does not look like a sqlite3 problem to me at all, but you clearly spent a lot more time on it12:00
Wizzupmaybe in how gnome folks use it12:00
freemangordonso it is either sqlite or how tracker uses it12:00
freemangordonright12:00
freemangordonbut I see nothing special in how they open the db12:01
Wizzupstill it would be good to know where this row comes from12:01
freemangordonresult = sqlite3_open_v2 (db_interface->filename, &db_interface->db, mode | SQLITE_OPEN_NOMUTEX, NULL);12:01
freemangordonthat's look fine to me12:01
Wizzupthe pragma's they run afterwards matter more12:01
freemangordon*that looks12:01
freemangordonyeah12:01
freemangordonlemme chech what they do12:01
freemangordon*check12:01
Wizzupis there any way to run tracker-miner-fs in a way where it *prints* what files it finds / wants analysed?12:02
Wizzupwell, I guess it does write that to the log (Creating new item)12:02
freemangordon'PRAGMA synchronous = OFF;'12:02
WizzupI've seen tracker-extract run a lot of pragma's, but tracker-miner-fs not so much, but I maybe did not see it restart as often12:03
Wizzupfreemangordon: yes, because they use journal mode and wal, I think that is fine12:03
freemangordonyou have to G_MESSAGES_DEBUG=all12:03
Wizzupdo you see tracker-miner-fs add any file before running that select, or not?12:04
WizzupI'm not sure yet how you know the problem isn't just within the tracker-miner-fs transaction/session12:04
freemangordonyes, I see adding many files12:05
freemangordonor rather it checks db for consistency on startup12:05
freemangordonlooks fine to me12:05
Wizzupalso in the odd behaviour where it keeps starting extract?12:05
freemangordonodd behaviour is because it thinks there is something to be extracted12:05
freemangordonbecause the query returns result12:05
Wizzupyes, but something has to be written to be db for that to return results12:06
freemangordonhttps://pastebin.com/fJ4iAxFx12:06
freemangordonpragmas12:06
Wizzupand I doubt that tracker-extract adds new rows by itself, it should just update them with data, no?12:06
freemangordonhmm, maybe you are right12:06
Wizzupthose are the same pragma's as with tracker-extract at least12:06
freemangordonI have a full startup log, lemme see where this uuid comes form12:07
freemangordonhmm, maybe I shall delete most of the music, to speed-up the process12:17
freemangordonWizzup: ok, what I think happens is:12:22
freemangordonminer crawls all directories and inserts files that are extract subject12:23
freemangordonthe tracker-extract (or tracker-store) deletes those records once processed12:23
freemangordonsomehow a stale data for a deleted record remains in the miner sqlite connection12:24
rafael2k_hi all! I'm flying to conference today, but on Sunday I'm back and can leave the PPP on and accessible over the VPN for anyone wanting to help with h-d and everything else!12:30
freemangordonrafael2k_: thanks, I guess Wizzup will unpack his device finally :)12:31
rafael2k_eheheh12:31
rafael2k_cool!12:31
freemangordonI am kind of reluctant to do development on a device noone has physical access to12:32
rafael2k_I'll use openvpn, as I already settled it up, but running to the airport now12:32
freemangordonin theory I can have a runaway process that will melt the cores12:32
rafael2k_ehehehe12:32
freemangordonno, really12:32
rafael2k_I always do this when compiling the kernel12:32
rafael2k_:P12:32
freemangordonhehe12:32
rafael2k_and now with 6 cores...12:32
freemangordonbut you know it will finish at some points12:33
freemangordonrunaway process will most-probably destroy the device if left running for sevral days12:33
Wizzupback12:33
rafael2k_btw, libcamera properly list the cameras, and Pinchart will add to libcamera some tuning files for the rk1isp12:33
rafael2k_for photo / video, PPP will work well12:34
Wizzupfreemangordon: I don't know if tracker-extract deletes or update records but this sounds correct12:34
freemangordondeletes them12:34
Wizzupok12:35
freemangordontracker sparql -q 'SELECT * WHERE { ?u a nfo:FileDataObject}'12:35
Wizzupwell, tracker-extract  doesn't find the record, do they even use the same kind of queries?12:35
freemangordonwho cares? the same query gives no results from the cli12:35
freemangordonexecute ^^^ query12:35
Wizzupright, it's not in the same transaction12:35
Wizzupin any case it sounds like what you said above seems correct, but we still might want to know what this stale row is, where it comes from12:36
Wizzupit doesn't seem like tracker-extract is doing anything wrong, is what I mean, it just starts, sees nothing, stops12:36
freemangordonit comes from the initial inserts in the miner12:36
freemangordonexactly12:36
sunshaviBut when you open from the cli. Yo do not use sqlite3_open_v2 which is multi-thread. so conn is different12:36
freemangordonand miner restarts it because it thinks extract has something to do :)12:36
freemangordonsunshavi: so? it is still the same db12:37
Wizzupfreemangordon: but it seems to be wrong in this belief, so I'm wondering where this stale record comes from12:37
rafael2k_* Kieran Bingham will help with the ISP side12:37
freemangordonWizzup: exactly12:37
Wizzuprafael2k_: great @ libcamera (sorry caught up in this tracker issue chat :) )12:37
Wizzupfreemangordon: the uuid doesn't show which one it is in the logs I guess?12:37
freemangordonminer does not print uuds12:38
WizzupI'm kind of surprised that the table with the record wouldn't contain the *path* of the file, like how does tracker-extract what to do12:38
rafael2k_Wizzup, : ))  I wish good luck with it!12:38
freemangordonbut it does not matter eitehr12:38
freemangordonWizzup: it does12:38
Wizzuplike, how does tracker extract know what to do with it*12:38
Wizzupfreemangordon: ok12:38
freemangordonit does know, while the record is still there12:38
Wizzupcan we get the path from tracker-miner-fs when it runs this query?12:38
freemangordonsparql -q 'SELECT * WHERE { ?u a nfo:FileDataObject; nie:url ?url. FILTER (?u = urn:uuid:14ec2b01-d9ad-432a-a4ce-12474c0f1f97)}'12:38
freemangordonwas giving me file:///home/user/Xorg.0.log, urn:uuid:14ec2b01-d9ad-432a-a4ce-12474c0f1f9712:39
freemangordonbefore extract did his job12:39
freemangordonand none after it did12:39
freemangordonthat's why I said extract (or store) deletes the records that were put there by the miner12:39
Wizzupok, so this record is there saved in the db before tracker runs, regardless of whether you look from miner-fs or not?12:39
Wizzupor is this just -a- record, not the stale one ? (sorry I'm bad at matching uuids :D )12:40
freemangordonwhat is 'tracker' in this context?12:40
Wizzuptracker-extract, my bad12:40
freemangordonyes, just a record12:40
Wizzupok12:40
freemangordona valid one12:40
Wizzupbtw, just to double check, when I run:12:40
Wizzup$ tracker sparql -q 'SELECT * WHERE { ?u a nfo:FileDataObject}'12:40
WizzupResults: None12:40
freemangordonmhm12:40
WizzupI get the above, and that is not a 1 row result, yeah?12:40
freemangordoneverything is deleted12:40
Wizzupok12:40
freemangordonno, this is empty result12:41
Wizzupok12:41
freemangordonin the same time miner gets results for teh same (lets assume) query12:41
freemangordonso, to me it is sqlite cached something12:41
freemangordongive me 2 minutes to debug something12:42
Wizzupk, it might be that miner-fs doesn't finish/flush it's work12:42
freemangordonoh, still extracting ::(12:42
Wizzupbefore it does a search itself, but even that should work12:42
freemangordonhmm, wait12:42
freemangordonmaybe it is on the opposite12:42
freemangordonit has it in its cache, but it is not flushed to the db12:43
freemangordonis that possible?12:43
sunshaviYes that could be possible. before doing commit or rollback12:44
freemangordondoes it read uncommited?12:44
Wizzupfreemangordon: sure it is possible, but then it would flush a stale record to the db, that is also weird12:45
Wizzupat least I am still under the assumption that tracker-miner-fs doesn't actually have any work that needs done12:47
freemangordonWizzup: seems sqlite does "read uncommited" if a query is in the same transaction12:47
Wizzupyes, that is how it should be12:47
WizzupI don't know if tracker-miner-fs does these things though12:48
freemangordonso it is possible that miner does not commit the last transaction12:48
Wizzup(just didn't look at the src)12:48
Wizzupyup, that would be hella stupid, but it's possible12:48
freemangordonso it will see the data, while noone else will12:48
freemangordonwell, I don;t see any other plausible option12:48
Wizzupyes, that would make sense if we can't see it, even it tracker-extract does not run12:49
freemangordoneither that or sqlite3 is FUBAR :)12:49
Wizzupfreemangordon: can we change tracker-miner-fs and print the rows it finds?12:49
Wizzupbefore it runs tracker-miner-fs ?12:49
Wizzuper12:49
Wizzuptracker-extract12:49
freemangordonas I said, I already did12:49
freemangordonwhat do you mean "rows"?12:49
freemangordonurl?12:49
Wizzupto get the filename12:49
Wizzupyeah12:49
freemangordonwhat data do you need?12:49
freemangordonok, sec12:49
Wizzupbtw, this is super unrelated, but probably also want to tweak the actual files that tracker looks at, there is the gsettings ignore rules and rules in /usr/share/tracker-miners/extract-rules12:50
Wizzupbut we probably*12:50
freemangordonright12:50
sunshaviwhat tracker has to do with an X log file?12:52
Wizzupnevermind the log file, that is a red herring12:52
Wizzupsunshavi: GNOME tracker reads all kinds of files in various places, including the Xorg.0.log file in /home/user12:52
freemangordonthis is a text file12:53
freemangordonthere is no reason for it to not be indexed12:53
Wizzupfreemangordon: tracker indexes text files for full text search12:53
Wizzupwhich actually is quite awesome12:53
Wizzupso there is a reason for it, but not for OMP12:53
Wizzupit also extracts text from PDFs12:54
sunshavibut that log file should have nothing interesting. It could be important If You are searching for an ERR perhaps12:54
Wizzupsunshavi: again, red herring, just nevermind12:54
Wizzup:)12:54
freemangordonWizzup: I didn;t say it is bad12:57
freemangordon"there is no reason for it to not be indexed"12:57
Wizzupmisread :)12:57
sunshaviexcluding that file does not help?12:57
Wizzupsunshavi: *red herring*, it's not related to the problem :D12:57
freemangordonsunshavi: this file has nothing to do12:57
freemangordonperiod :)12:57
sunshaviperhaps on the miner's codepath a commit or rollback have been forgotten?12:59
Wizzupperhaps, we'll find out12:59
freemangordonWizzup: runnung the new miner that will dump all the 'stale' rows13:01
freemangordonthe fuck13:02
freemangordonI deleted most of the music and now it works ok13:02
freemangordon(tracker-miner-fs:17937): Tracker-DEBUG: 14:01:49.455: Not starting extractor. Nothing to do.13:02
freemangordonok, lemme add data back13:02
Wizzuplol: 06 Dec 2023, 13:03:51: Tracker: Processing file 'file:///usr/share/applications'...13:04
Wizzup06 Dec 2023, 13:04:15: Tracker: Creating new item 'file:///usr/share/applications/hildon-status-menu/ham-notifier.desktop'\13:04
Wizzupnot sure why but ok13:04
freemangordonit indexes applications as well13:05
freemangordonI think they disabled that in 3.013:05
Wizzup$ gsettings get org.freedesktop.Tracker.Miner.Files index-recursive-directories13:05
Wizzup['&DESKTOP', '&DOCUMENTS', '&MUSIC', '&PICTURES', '&VIDEOS', '/mnt/sd/']13:05
Wizzupit shouldn't I think, but ok13:05
freemangordonhmm, looks like I am not hitting the bug with the modified query :(13:13
freemangordonor I still don;t have enough music13:13
freemangordonWizzup: Result [0]: urn:uuid:d8630ae1-784b-4112-8e9b-47edd6060d8c, file:///home/user/MyDocs/.sounds/Blind%20Guardian/(1986-1987)%20-%20Lucifer's%20Heritage%20E.P.'s/(1987)%20-%20Battalions%20Of%20Fear/05%20-%20Run%20For%20The%20Night.mp313:21
freemangordonhmm, wait13:22
freemangordonthis is the first time after restart so it is normal13:22
Wizzupmhm13:23
freemangordonthe second time it still says "(tracker-miner-fs:18865): Tracker-DEBUG: 14:21:25.565: Not starting extractor. Nothing to do."13:23
freemangordonlemme add more music13:23
freemangordonmaybe the changed query made it run properly, dunno13:24
freemangordonI added   OPTIONAL {?u nie:url ?url} in the select clause13:24
Wizzupit might take a bit of time to get into this problem13:27
freemangordonlooks like, yeah13:27
WizzupI'm also reindexing my files, even though I did the same yesterday but some how it got lost I guess13:29
freemangordonwill take some time until music gets copied over ssh13:31
Wizzupfreemangordon: hm pvr on my d4 is unhappy, any debug I can get for you?13:38
freemangordonelaborate on 'unhappy'13:38
Wizzupscreen is stuck, nothing happens, dmesg is filled with repeating messages13:38
freemangordonI don;t think I can do anything about it13:38
Wizzupinterestingly it starts with:13:39
WizzupDec  6 13:07:54 localhost kernel: [130585.629302] PVR_K: User requested SGX debug info13:39
freemangordonin the newer blobs they say reset works properly13:39
freemangordonno idea what that means13:39
Wizzupxorg log says:13:39
Wizzup[130585.607] (EE) OMAP(0): ERROR: waitForBlitsCompleteOnDeviceMem: PVR2DQueryBlitsComplete failed with error code: -8 (Blit not complete)13:40
Wizzup[130588.704] (EE) OMAP(0): ERROR: sgxCopyNextBatch: PVR2DBlt failed with error code: -2 (Device unavailable)13:40
Wizzup[130589.204] (EE) OMAP(0): ERROR: waitForBlitsCompleteOnDeviceMem: PVR2DQueryBlitsComplete failed with error code: -8 (Blit not complete)13:40
freemangordonsgx crashed13:40
freemangordonthis shall be handled in the driver/blobs13:40
freemangordonbut it is not13:40
freemangordonin our version of the blobs that is13:41
Wizzupok13:41
WizzupI'll restart the device when tracker finishes13:41
Wizzupbrb13:41
freemangordonsee the commit message https://git.ti.com/cgit/graphics/omap5-sgx-ddk-um-linux/commit/?h=1.17.4948957/mesa/glibc-2.35&id=337832df9662f0842ce4704342c0e42acab2220913:42
freemangordon"Hardware recoveries work now,..."13:42
freemangordonok, we I hit the bug again13:44
freemangordonResult [0]: urn:uuid:9b88e7a9-30de-4153-8b60-e8b6ea7ed1a3, file:///home/user/MyDocs/.sounds/Hammerfall/Renegade/06%20-%20The%20Way%20Of%20The%20Warrior.mp313:44
freemangordonseems tracker does not like Hammerfall :)13:44
freemangordonhave to join a mtg, bbl13:46
Wizzupfreemangordon: looks like my phone is in this state again13:58
Wizzupthe tracker one13:58
freemangordonmine as well14:00
Wizzupwhen I then run tracker daemon -k, it loses -all- the data14:00
freemangordonbut will continue debuggin when have time14:00
Wizzupsigh14:00
freemangordonWizzup: seems tracker misuses sqlite API15:25
freemangordonthey use   sqlite3_prepare_v2() and  deprecated sqlite3_expired()15:26
freemangordonalso, no check is made for the result of sqlite3_reset()15:28
sunshaviperhaps it is old code never updated15:33
freemangordonno, there is a note there15:33
sunshavimmm15:33
sunshaviwhat about checking the res of sqlite3_reset(). Does that helps?15:38
freemangordonhehe, that's what I am trying to do now15:41
freemangordonwaiting for it to be hit15:41
sunshaviok.15:43
freemangordonhmm, now  sqlite3_step does not return an error15:47
freemangordonit returns SQLITE_ROW15:48
sunshaviaccording to documentation two alternatives sqlite3_reset | sqlite3_step15:54
freemangordonsunshavi: seems to be uncommitted data15:55
sunshavithen commit first?15:56
freemangordonhmm?15:56
sunshavior rollback (more context would be needed to reach the right alternative)15:56
freemangordonwhere?15:56
sunshavithat uncommited data should be flushed? then commit otherwise rollback15:57
freemangordonwhere?15:57
freemangordonI know what has to be done15:57
sunshaviat which point in the source code commit is going to be hit?. this data never flushes cos commit never happens. is this a kind of daemon?. Yes it is cos it reuses the connection. could I do a clone of this code to see the specific line we are talking about?16:08
Wizzupsunshavi: he's telling you that he knows what you know, but he needs to find the place in the tracker code and find where it is not flushed16:08
freemangordonsunshavi: believe me, I know the theory16:09
sunshaviok16:09
freemangordonthe code in question is tracker16:09
freemangordonno idea where in it the bug is16:09
freemangordonWizzup: ok, at least one issue - tracker reset -r deletes sqlite db, but mafw never knows about that16:16
freemangordonso it keeps the deleted files openend :)16:16
Wizzupyes that is a problem that I also experience16:16
Wizzupbut it's not the cause for our troubles16:16
freemangordonone of them being some file called $something-shm16:16
freemangordoncould be16:17
freemangordonbut I will try to fix that first16:17
WizzupI had the trouble occur when mafw source tracker was stopped16:17
freemangordonyes, I know16:17
freemangordonI had it stopped here as well16:17
freemangordonbut we have multiple issues16:17
freemangordonanyway, have to run now, bbl16:18

Generated by irclog2html.py 2.17.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!