Ubuntu: Multi-threading in application designed in quickly


I have a tree which I populate with a file list from my computer on the click of a button. While the program is populating the tree the whole GUI hangs.

Is there a way to populate the tree in another thread so the tree gets populated as the files are found and not when everything has already been added? Or any other ideas?

This is what it looks like:

Screenshot of App

I am populating the tree with this code:

def traversePaths(self, path, parent):          files = os.listdir(path)          files.sort()          for myfile in files:              if myfile[0] != ".":                  if os.path.isdir(path + os.sep + myfile):                      current = self.filelist.append(parent,[self.dirIcon,self.dirIcon,self.dirOpenIcon,myfile,"DD",True,True,3])                      self.traversePaths(path + os.sep + myfile, current)                  else:                      current = self.filelist.append(parent,[self.fileIcon,self.dirIcon,self.dirOpenIcon,myfile,"AA",True,True,3])  

which is executed on a button click:

def on_refreshbutton_clicked(self, button):      self.traversePaths(self.path.get_filename(), None)  

I don't think threading.Thread works because of the gtk thread for the gui and cannot find gi.repository.Gtk api's on the subject

Any ideas?


When asking programming questions, it is always best to provide a minimal working example and not just code snippets.

Without a working example to play with, I have the following suggestions:

  1. Use a thread to walk the directory tree and put your results in a second separate treemodel (not connected to a treeview). This means you have two treemodels, one connected (which gets rendered to the screen) and one not connected (which therefore does not get rendered)
  2. Add a timeout of a few seconds, which does calls a function that detaches and deletes the first treemodel, copies the second treemodel to a new one, which now serves as the "thread" treemodel and attaches the second tree model.

The reason why you see the GUI hanging is that the file walker takes very long and every time you add a file, a lot of overhead in Gtk is called. By adding a full treemodel, this overhead is only called one time. And by using a thread for the file walking, your GUI stays responsive.

As for the threading part in Gtk, you might want to look here: https://stackoverflow.com/questions/8120860/python-doing-some-work-on-background-with-gtk-gui

A couple of notes to your code:

  • Python has a build-in file walker, which might be faster but is certainly shorter than your code: os.walk
  • If you want to use your code instead, remember that Python has a build-in recursion limit. Depending on the size of your file system, you might want to replace the recursion with a trampoline like construct


Here's a complete example that updates tree concurrently using a single TreeStore:

#!/usr/bin/python  import os  import threading  import time  from itertools import cycle    from gi.repository import GObject, Gtk  GObject.threads_init()  # all Gtk is in the main thread;                          # only GObject.idle_add() is in the background thread      HEARTBEAT = 20   # Hz  CHUNKSIZE = 100  # how many items to process in a single idle_add() callback      def chunks(seq, chunksize):      """Yield N items at a time from seq."""      for i in xrange(0, len(seq), chunksize):          yield seq[i:i + chunksize]      class TreeStore(Gtk.TreeStore):      __gtype_name__ = 'TreeStore'        def __init__(self, topdir, done_callback=None):          Gtk.TreeStore.__init__(self, str)  # super() doesn't work here            self.path2treeiter = {topdir: None}  # path -> treeiter          self.topdir = topdir          self.done_callback = done_callback          self._cv = threading.Condition()            t = threading.Thread(target=self._build_tree)          t.daemon = True          t.start()  # start background thread        def _build_tree(self, _sentinel=object()):          # executed in a background thread          cv = self._cv          p = self.path2treeiter          for dirpath, dirs, files in os.walk(self.topdir):              # wait until dirpath is appended to the tree              cv.acquire()              while p.get(dirpath, _sentinel) is _sentinel:                  cv.wait()              parent = p[dirpath]              cv.release()                # populate tree store              dirs[:] = sorted(d for d in dirs                               if d[0] != '.')  # skip hidden dirs              for chunk in chunks(dirs, CHUNKSIZE):                  GObject.idle_add(self._appenddir, chunk, parent, dirpath)                for chunk in chunks(sorted(files), CHUNKSIZE):                  GObject.idle_add(self._appendfile, chunk, parent)          GObject.idle_add(self.done_callback)        def _appenddir(self, chunk, parent, dirpath):          # executed in the main thread          self._cv.acquire()          p = self.path2treeiter          for d in chunk:              p[os.path.join(dirpath, d)] = self.append(parent, [d])          self._cv.notify()          self._cv.release()        def _appendfile(self, chunk, parent):          # executed in the main thread          for f in chunk:              self.append(parent, [f])      class Window(Gtk.Window):      __gtype_name__ = 'Window'        def __init__(self, topdir):          super(Window, self).__init__(type=Gtk.WindowType.TOPLEVEL)          self.__start_time = time.time()          self.__title = 'GTK Tree MultiThreading Demo'          self.set_title(self.__title)          self.set_default_size(640, 480)            # create tree          tree_store = TreeStore(topdir, self._on_tree_completed)          tree_view = Gtk.TreeView()          tree_view.set_model(tree_store)            cell = Gtk.CellRendererText()          tree_view.append_column(Gtk.TreeViewColumn(topdir, cell, text=0))          scrolled_window = Gtk.ScrolledWindow()          scrolled_window.add(tree_view)          self.add(scrolled_window)            # update title to show that we are alive          self._update_id = GObject.timeout_add(int(1e3 / HEARTBEAT),                                                self._update_title)        def _on_tree_completed(self):          if self._update_id is None:              return          # stop updates          GObject.source_remove(self._update_id)          self._update_id = None          self.set_title('%s %s %.1f' % (self.__title, ' (done)',                                         time.time() - self.__start_time))        def _update_title(self, _suff=cycle('/|\-')):          self.set_title('%s %s %.1f' % (self.__title, next(_suff),                                         time.time() - self.__start_time))          return True  # continue updates      win = Window(topdir=os.path.expanduser('~'))  win.connect('delete-event', Gtk.main_quit)  win.show_all()  Gtk.main()  

As @xubuntix said each treestore.append() is expensive in this case. See whether it is acceptable for you.

Note:If u also have question or solution just comment us below or mail us on toontricks1994@gmail.com
Next Post »