| arno11 | Wizzup: i found why conversations UI is randomly so slow to start: the root cause is the tracker, and it affects most qt5 apps. | 09:34 |
|---|---|---|
| arno11 | disabling the tracker with 'tracker daemon -t' speeds up things a lot on the n900 | 09:35 |
| arno11 | (i noticed that when you launch any qt5 app, tracker-store starts immediately and uses lot of cpu for a couple of sec) | 09:42 |
| arno11 | with tracker disabled, qt5 apps start really faster | 09:44 |
| Wizzup_ | huh, why would tracker do that | 11:19 |
| arno11 | Wizzup: really no idea :( | 12:11 |
| arno11 | but without tracker, qt5 apps start in 2-5 sec | 12:12 |
| Wizzup_ | maybe it's somehow atime related | 12:13 |
| Wizzup_ | do we mount with atime? | 12:13 |
| Wizzup_ | relatime | 12:13 |
| Wizzup_ | probably dbus can tell us | 12:15 |
| Wizzup_ | I haven't noticed this before, but I'll try to reproduce | 12:15 |
| arno11 | ok | 12:15 |
| arno11 | i haven't noticed this before too | 12:16 |
| Wizzup_ | freemangordon: shouldn't we use noatime? | 12:19 |
| Wizzup_ | or at least lazyatime | 12:19 |
| Wizzup_ | I mean relatime isn't necessarily our problem here wrt tracker, but I don't see why we would keep it enabled since we're mostly on nand/emmc | 12:21 |
| uvos | Wizzup_: tracker uses some cpu whenever something in the tracked folders changes which it watches via inotifywait | 13:24 |
| uvos | it seams that in the default configuration its watching $HOME | 13:24 |
| uvos | so maybe qt touches alot of config files | 13:25 |
| uvos | regarless i dont think this antivirus like behavior is sane on a phone and we should restict its scanning tightly to some media directory | 13:25 |
| Wizzup_ | uvos: either that or atime | 13:30 |
| Wizzup_ | but I agree probably scanning $HOME is not great | 13:30 |
| Wizzup_ | I don't think it does that recursively though | 13:30 |
| uvos | it somehow found random media in downloads for me | 13:30 |
| uvos | so i think it dose recusevly | 13:30 |
| uvos | or maybe thats from XDG places spec | 13:31 |
| Wizzup_ | you can find in gsettings what it searches | 13:33 |
| Wizzup_ | and yeah it does search some xdg docs/download stuff | 13:33 |
| freemangordon | yeah, maybe we shall exclude $HOME | 16:18 |
| freemangordon | Wizzup_: why would noatime affect tracker performance? | 16:19 |
| freemangordon | do you think it gets inotified on a file being read? | 16:20 |
| freemangordon | Wizzup_: and yes, I also think tracker does recursive scanning | 16:20 |
| Wizzup_ | freemangordon: I don't know what inotify does with access time changing | 16:24 |
| Wizzup_ | freemangordon: | 16:24 |
| Wizzup_ | $ gsettings get org.freedesktop.Tracker.Miner.Files index-recursive-directories | 16:24 |
| Wizzup_ | ['&DESKTOP', '&DOCUMENTS', '&MUSIC', '&PICTURES', '&VIDEOS', '/mnt/sd/'] | 16:24 |
| Wizzup_ | $ gsettings get org.freedesktop.Tracker.Miner.Files index-single-directories | 16:25 |
| Wizzup_ | ['&DOWNLOAD'] | 16:25 |
| Wizzup_ | I don't see $HOME there | 16:25 |
| Wizzup_ | (/mnt/sd is what I added obviously) | 16:25 |
| Wizzup_ | I wonder if we awnt to play with 'throttle' and/or 'index-on-battery' perhaps | 16:26 |
| Wizzup_ | we could also set enable-monitors to false, but I think the inotify thing is mostly ince | 16:26 |
| freemangordon | ok, maybe we shall start miner/extractor from the shell with verbose log enabled, to check if they really do something on qt app starting | 16:27 |
| freemangordon | I am suspicious about the '...is randomly so slow...' part | 16:29 |
| freemangordon | it either runs every time or not | 16:29 |
| Wizzup_ | mhm | 16:29 |
| freemangordon | that could be just a coincidence that tracker runs when he tried to start an app | 16:29 |
| freemangordon | actually conversations is qml, and afaik qml caches shaders | 16:37 |
| freemangordon | that might somehow interfere | 16:37 |
| Wizzup_ | I doubt those are in any of the dirs in index-recursive-directories | 16:38 |
| arno11 | freemangordon: i doubt it is a coincidence. when i say 'randomly', that's because sometimes conversations started in 8, 10, 15 or even 20 sec. | 20:06 |
| arno11 | now that's 2, 4, 8 sec max | 20:06 |
| freemangordon | I see | 20:08 |
| freemangordon | arno11: could you start tracker processes from console with verbose debug enabled to see what's going on? | 20:08 |
| arno11 | sure | 20:09 |
| arno11 | tracker daemon -s --set-log-verbosity=debug ? | 20:11 |
| freemangordon | not sure | 20:11 |
| freemangordon | there are several processes | 20:12 |
| freemangordon | lemme check | 20:12 |
| arno11 | ok | 20:12 |
| freemangordon | arno11: try first with /usr/libexec/tracker-extract -v 3 | 20:17 |
| arno11 | ok | 20:17 |
| freemangordon | and then immediately run some qt process | 20:17 |
| arno11 | ok | 20:17 |
| freemangordon | as it will exit after 10 seconds of inactivity | 20:18 |
| freemangordon | if nothing useful, then stop tracker-miner-fs and then start it by hand with /usr/libexec/tracker-miner-fs -n -v 3 | 20:19 |
| freemangordon | hmm, seems after all $HOME is monitored | 20:21 |
| freemangordon | (tracker-miner-fs:5769): Tracker-DEBUG: 21:37:32.607: Received monitor event:3 (G_FILE_MONITOR_EVENT_CREATED) for file:'file:///home/user/orcexec.NDMv16' | 20:21 |
| arno11 | (dinner time) bbiab | 20:23 |
| Wizzup_ | it might be monitored but it shouldn't act on it I think | 20:23 |
| freemangordon | why it is monitored if we dont act on it? | 20:24 |
| freemangordon | Tracker-Message: 21:41:33.921: Setting up directories to iterate from config (IndexSingleDirectory) | 20:25 |
| freemangordon | Tracker-Message: 21:41:33.921: Adding:'/home/user' | 20:25 |
| freemangordon | Added monitor for path:'file:///home/user', total monitors:1 | 20:26 |
| freemangordon | so $HOME *is* monitored | 20:26 |
| freemangordon | and it seems qt creates a file there | 20:26 |
| freemangordon | (tracker-miner-fs:4012): Tracker-DEBUG: 21:44:33.736: Processing file 'file:///home/user/orcexec.THNr5D'... | 20:27 |
| freemangordon | why the hell creates file in $HOME?!? | 20:27 |
| freemangordon | *qt creates | 20:27 |
| Wizzup_ | what files are created there? | 20:30 |
| Wizzup_ | btw, I don't think it makes sense for tracker to be this realtime with its indexing | 20:30 |
| Wizzup_ | can you check your index single directory gsettings and see what they say? | 20:30 |
| freemangordon | https://askubuntu.com/questions/336552/what-are-orcexec-files | 20:30 |
| freemangordon | this ^^^ | 20:31 |
| Wizzup_ | freemangordon: what about your gsettings? | 20:35 |
| Wizzup_ | do they list $HOME | 20:35 |
| Wizzup_ | gsettings get org.freedesktop.Tracker.Miner.Files index-single-directories | 20:37 |
| Wizzup_ | and | 20:37 |
| Wizzup_ | gsettings get org.freedesktop.Tracker.Miner.Files index-recursive-directories | 20:37 |
| freemangordon | user@devuan:~$ gsettings get org.freedesktop.Tracker.Miner.Files index-single-directories | 20:38 |
| freemangordon | ['$HOME', '&DOWNLOAD'] | 20:38 |
| freemangordon | arno11: ^^^ could you check on n900 | 20:38 |
| freemangordon | https://gstreamer.freedesktop.org/projects/orc.html | 20:40 |
| freemangordon | who with his right mind creates temp files in $HOME?!? | 20:41 |
| Wizzup_ | freemangordon: wow why is $HOME in there | 20:41 |
| Wizzup_ | I mean for single I guess it makes sense | 20:41 |
| freemangordon | ies, it is in single | 20:41 |
| Wizzup_ | not sure actually | 20:41 |
| freemangordon | *yes | 20:41 |
| Wizzup_ | in any case, maybe arno can remove the entry from his gsettings and see if it solves the problem | 20:42 |
| freemangordon | I don;t think it should be there | 20:42 |
| freemangordon | mhm | 20:42 |
| freemangordon | this is the default I guess | 20:42 |
| freemangordon | we can fix it | 20:42 |
| Wizzup_ | honestly to me it seems like we should set a 'crawling-interval' and disable enable-monitors | 20:42 |
| Wizzup_ | and maybe see that the 'throttle' key does | 20:42 |
| freemangordon | no, please, why? | 20:42 |
| Wizzup_ | ( gsettings list-keys org.freedesktop.Tracker.Miner.Files ) | 20:43 |
| Wizzup_ | why, well, because it seems to in general wreak havoc on n900 | 20:43 |
| freemangordon | if we don;t monitor $HOME it should be better | 20:44 |
| Wizzup_ | sure | 20:44 |
| Wizzup_ | hopefully | 20:44 |
| Wizzup_ | but this could very well be why things are slow(er) during boot too | 20:44 |
| freemangordon | and I prefer to have my music files indexed ASAP, not in 2 hours | 20:44 |
| freemangordon | mhm | 20:46 |
| Wizzup_ | https://gitlab.gnome.org/GNOME/tracker-miners/-/issues/236 | 20:47 |
| Wizzup_ | throttle is set to 0 by default (bad) | 20:48 |
| Wizzup_ | I suggest we set initial-sleep to something like 600 (seconds), it's currently 15 | 20:48 |
| freemangordon | agree | 20:48 |
| freemangordon | well, 600 is kinda extreme | 20:49 |
| Wizzup_ | it's only 10 minutes | 20:49 |
| Wizzup_ | but fine, something high enough at least | 20:49 |
| freemangordon | yes, lets make it 2 minutes initially | 20:49 |
| Wizzup_ | I think that's not enough for n900 to boot | 20:49 |
| freemangordon | we can increase it further | 20:49 |
| Wizzup_ | I'd rather make it long first and then decrease if we find the boots are much better | 20:50 |
| freemangordon | right, but combined with throttle will help | 20:50 |
| Wizzup_ | it might also make sense to set this to true: | 20:50 |
| Wizzup_ | gsettings get org.freedesktop.Tracker.Extract wait-for-miner-fs | 20:50 |
| Wizzup_ | per https://askubuntu.com/questions/1258757/tracker-miner-fs-is-taking-90-of-cpu | 20:51 |
| Wizzup_ | also, the issue report I mentioned mentions tracker3, which I think we are not on yet | 20:51 |
| freemangordon | also, see https://github.com/maemo-leste-upstream-forks/tracker-miners/blob/maemo/chimaera-devel/data/org.freedesktop.Tracker.Miner.Files.gschema.xml#L141 | 20:51 |
| Wizzup_ | I guess I must have removed it myself | 20:53 |
| freemangordon | mhm | 20:53 |
| freemangordon | but still, I don;t see why temp files shall be created in $HOME | 20:53 |
| Wizzup_ | I can see why, many systems have some tmpfs set to noexec | 20:54 |
| Wizzup_ | whereas that is unlikely for something in home, but you'd still expect it to maybe go in .cache or something | 20:55 |
| freemangordon | mhm | 20:55 |
| Wizzup_ | arno11: let us know when you're back, I think changing some settings will make all the difference | 20:55 |
| freemangordon | I'd recommend to try changing throttle first | 21:03 |
| Wizzup_ | and remove home, initial sleep.. | 21:10 |
| arno11 | Wizzup: back | 21:10 |
| freemangordon | Wizzup_: https://cgit.freedesktop.org/gstreamer/orc/tree/orc/orccodemem.c#n305 | 21:11 |
| freemangordon | it should create in XDG_RUNTIME_DIR, not in HOME | 21:12 |
| arno11 | i'm ready if you want me to change some settings | 21:16 |
| freemangordon | Wizzup_: ^^^ | 21:19 |
| Wizzup_ | hi | 21:23 |
| Wizzup_ | arno11: maybe try to set | 21:24 |
| Wizzup_ | gsettings set org.freedesktop.Tracker.Miner.Files throttle 20 | 21:24 |
| Wizzup_ | gsettings set org.freedesktop.Tracker.Miner.Files initial-sleep 300 | 21:24 |
| Wizzup_ | gsettings set org.freedesktop.Tracker.Extract wait-for-miner-fs true | 21:24 |
| Wizzup_ | and maybe remove the HOME entry from: | 21:24 |
| Wizzup_ | gsettings get org.freedesktop.Tracker.Miner.Files index-single-directories | 21:24 |
| arno11 | ok | 21:25 |
| Wizzup_ | you could skip the last part for now | 21:26 |
| Wizzup_ | in any case, I hope this will help with boot times and converstionst startup | 21:26 |
| Wizzup_ | freemangordon: I think we set XDG_RUNTIME_DIR, | 21:26 |
| Wizzup_ | don't we? | 21:26 |
| freemangordon | yes, we do | 21:26 |
| freemangordon | ok, I am a bit lost | 21:26 |
| freemangordon | after a restart those temp files are no longer created in $HOME | 21:26 |
| arno11 | Wizzup_: ok, do i need to run other stuff or can i reboot now ? | 21:29 |
| freemangordon | hmm, it seems gst creates them | 21:32 |
| Wizzup_ | arno11: reboot is fine | 21:33 |
| arno11 | ok let's go | 21:34 |
| freemangordon | Wizzup_: tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=401312k,mode=755) | 21:35 |
| freemangordon | '/run' has noexec set | 21:36 |
| Wizzup_ | as expected | 21:36 |
| Wizzup_ | brb | 21:36 |
| freemangordon | well, but XDG_RUNTIME_DIR=/run/user/1000 | 21:36 |
| freemangordon | and orc does fallback to $HOME | 21:37 |
| Wizzup | well we don't want to remove noexec | 21:39 |
| freemangordon | why? https://github.com/systemd/systemd/issues/4081 | 21:40 |
| freemangordon | the question "where JIT compilers are supposed to write their output" is valid | 21:41 |
| Wizzup | runtime dir is for pid files and stuff | 21:42 |
| freemangordon | no, it is user specific | 21:42 |
| freemangordon | run/user/$UID | 21:42 |
| freemangordon | as is $HOME | 21:42 |
| freemangordon | read the issue ^^^ | 21:42 |
| Wizzup | yeah I don't agree with lennart (as usual) | 21:43 |
| freemangordon | ok, so, to your understanding, what is the correct place to put JIT compiler output? | 21:44 |
| Wizzup | XDG provides none | 21:44 |
| Wizzup | https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html | 21:44 |
| Wizzup | I'd probably use $XDG_CACHE_HOME | 21:45 |
| freemangordon | $XDG_RUNTIME_DIR - ...The directory MUST be on a local file system and not shared with any other system. The directory MUST by fully-featured by the standards of the operating system. More specifically, on Unix-like operating systems AF_UNIX sockets, symbolic links, hard links, proper permissions, file locking, sparse files, memory mapping, ... | 21:46 |
| Wizzup | I'm looking at openrc on my laptop and /run is not set noexec | 21:46 |
| Wizzup | I wonder what sets it on devuan then | 21:46 |
| freemangordon | Wizzup: we'll find that | 21:46 |
| freemangordon | but lets first agree on whether it should be noexec or not :) | 21:47 |
| Wizzup | note that none of the things mentioned above are hurt by noexec afaik | 21:47 |
| freemangordon | memory mapping is | 21:47 |
| freemangordon | because you cannot map with PROT_EXEC | 21:47 |
| Wizzup | yes, so specific memory mapping doesn't I suppose | 21:48 |
| Wizzup | in any case if gentoo openrc doesn't do it then I'm fine with not doing it on devuan | 21:48 |
| freemangordon | maybe elogind make that | 21:48 |
| Wizzup | that would be truly hilarious given lennarts stance | 21:48 |
| freemangordon | mhm | 21:49 |
| Wizzup | https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1991661 | 21:49 |
| Wizzup | my debian server also has it set (noexec) | 21:49 |
| freemangordon | my ubuntu as well (14.04) | 21:50 |
| Wizzup | I found it via https://askubuntu.com/questions/1432383/mounting-run-as-noexec/1433208 | 21:50 |
| Wizzup | in any case I think this is pretty minor and I do think that tracker should not trip up over programs making tmp files in home | 21:50 |
| Wizzup | I'm fine with it using home as scratch | 21:50 |
| freemangordon | so, you think tracker shall not monitor $HOME? | 21:51 |
| freemangordon | I am fine, but I wonder if we reinstall tracker whether it will change the setting | 21:52 |
| Wizzup | even if it does, it doesn't need to act *immediately* and hurt the ux | 21:52 |
| Wizzup | there's no reason to find say a mp3 file within a second | 21:52 |
| freemangordon | what is the acceptable time? | 21:52 |
| Wizzup | minutes or more imo | 21:52 |
| freemangordon | not for me :) | 21:52 |
| freemangordon | I WANT MY MUSIC NOW!!! | 21:53 |
| freemangordon | :D | 21:53 |
| Wizzup | I mean even now it takes many minutes for it show up in omp | 21:53 |
| Wizzup | since we can write them faster than we can analyse them | 21:53 |
| freemangordon | only on first boot/copy | 21:53 |
| Wizzup | I really see tracker as a background thing that should eventually figure things out but not immediately | 21:53 |
| freemangordon | then if you add more files it happens way faster | 21:53 |
| freemangordon | but really, wating 10 minutes for your just copied music to appear in the player is very broken UX | 21:54 |
| Wizzup | I think that is currently already the case, but I have a big music collection on it (100GB or so) | 21:55 |
| freemangordon | Wizzup: and you wait 10 minutes after boot on d4 for OMP to show the files? | 21:55 |
| Wizzup | why after boot? | 21:56 |
| Wizzup | it is available immediately of course | 21:56 |
| Wizzup | no additional mining is necessary | 21:56 |
| freemangordon | right, seems I misunderstood | 21:56 |
| arno11 | Wizzup: unfortunately boot time is still high and qt5 apps are still slow to start :( | 21:56 |
| freemangordon | arno11: could you do /usr/libexec/tracker-miner-fs -v 3 -n and start that qt app | 21:57 |
| freemangordon | to see if miner gets notified | 21:58 |
| freemangordon | perhaps you should killall tracker-miner-fs first | 21:58 |
| arno11 | ok (i already stopped trackers) | 21:59 |
| Wizzup | arno11: did you also remove monitoring HOME? | 21:59 |
| arno11 | Wizzup: nope | 22:00 |
| arno11 | i can try | 22:00 |
| arno11 | what is the command btw ? | 22:00 |
| Wizzup | I need a few mins to figure it out | 22:02 |
| Wizzup | I think just set with the values you want to keep in quotes | 22:02 |
| Wizzup | but maybe make sure that killing trackers indeed solves it | 22:02 |
| Wizzup | I didn't expect boot time not to be high, but maybe to at least be a bit better | 22:02 |
| arno11 | freemangordon: sorry, tracker miner fs returns too much things. i can't find anything relevant atm. (just my n900 around) | 22:13 |
| arno11 | Wizzup: ok | 22:14 |
| Wizzup | I'll try to look at my n900 as well | 22:14 |
| arno11 | Wizzup: i'm still not able to remove $HOME, need to check how exactly gsettings commands work lol | 22:34 |
| uvos | eh honestly i think it might be better to just disable inotify indexing entirely by default and provide a "scan for media" button in omp... | 22:46 |
| Wizzup | uvos: yeah maybe | 23:47 |
| Wizzup | it might also ultimately get better from tracker3 | 23:47 |
| Wizzup | I just think we should for now try to minimise the time we spend on it | 23:47 |
Generated by irclog2html.py 2.17.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!