Together Tip: Large Import Performance

As mentioned previously, I am committed to making Together perform really well, including with large libraries, as this is a common complaint from some people about apps of this sort (or so I keep hearing). As I find out about individual performance problems, I’m fixing them. It’s quite a challenge, people make incredible demands of these sorts of applications.

As I see it, the top priority here is to have Together perform well in everyday operations and it does, although I’ll mention some more improvements coming in that area below.

However, some people have criticised Together’s performance in importing a large number of files (anything from 5,000 to 20,000), so I’ll try to explain the problem here and give some tips on how to do that quickly and successfully.

Importing Large Numbers of Files

Together imports files in the background. This is useful for a number of reasons, not least as a way for you to use the Shelf without blocking the app with a progress sheet, and to keep the Shelf compatible with Spaces (showing a progress sheet on a window automatically makes that window’s space active, which is a real pain when you’re working in another application).

This background importing is also good because it means you can continue to work in the application while files are imported. However, there’s a downside in that, for every file added to the library, Together has to sort the list of files.

If the Library group (which shows all the files in Together) is selected and it already contains thousands of items, the second or so that it takes to sort the items adds considerably to the time it takes to import 10,000 files. And the more files there are in the library, the longer it will take to sort them. As such, large imports will get progressively slower.

There’s an easy answer to this: select a group or folder that is not affected by the large import and therefore doesn’t have to sort the incoming files. Using this method, I’ve imported 10,000 items in about 30 minutes, which is how long it took to copy them in the Finder.

In general, this is not an issue. How often do you import thousands of items a time? It’s typically a one-off operation performed when first using the software and most people don’t do it at all. That some people’s impression of the software is formed through this is unrepresentative and unfortunate. I can understand why they would feel frustrated.

Performance vs. Responsiveness

There is also the matter of performance vs. responsiveness. I have some changes lined up for the next version of Together in this area.

To give an example, somebody wrote to me yesterday to say that they got the spinning beachball when switching between some groups, but not others. It turns out the the first item in the slow group was an 8 megabyte PDF file (equates to about 200 pages). So, once the group was selected, Together would immediately start loading that PDF. Try opening an 8MB PDF in Preview and see how long it takes before it’s ready for you to start working with it.

In Together, most previews are loaded in the background and then displayed. This makes the app appear more responsive (i.e. you could scroll through many items or switch groups without a lag), but you don’t see the preview any quicker. Together also caches previews of some files so it’s quicker to revisit them.

I try to avoid these measures when possible because the background loading actually makes the app slower for some kinds of files and the caching increases the memory footprint (if you use too much memory, that will also slow the app down), but I will be taking this approach for large PDF files in the next version.

4 Responses to “Together Tip: Large Import Performance”

  1. JRP Says:

    Just wanted to say that I am a DevonThink Pro user. While Together will not replace DT for mass researching and data storage, I am trying Together along with Yojimbo, Journier, and EagleFiler. Have to say that so far I prefer Together. Clean UI, the addition of smart folders and spotlight search, and really like the ease of the drop and quick note feature. The one thing I do like better in EagleFiler is the visibility and ease of its tags. Also, Together’s animation, i.e., top menu bar appearing/disappearing during modifications to searches is disturbingly slow visually. I’d love to be able to speed that up. But these are small points. I really am enjoying putting the app through it’s paces, and am interested in how it performs as I continue to stuff data in.

  2. JRP Says:

    Dang. I just had to send another email. 1) Correcting an error in previous: It’s the tag implementation that Yojimbo uses that I was thinking of, and 2) I love the way that Together handles emails and address book imports. Much better than any of the others, and miles ahead of DT. Thanks for an excellent app…I may have to pop for this one if the rests of my tests prove out.

  3. zenrain Says:

    I also am / was a DTP user. I must say that when using DT I didn’t know what the big deal about missing spotlight functionality was. Now that I’m using Together, although I have my files imported into the database, it is incredibly nice to be able to have immediate access to them through a search even when Together is closed.

    I’ve said it before, and I’ll say it again, great app. Thanks!

  4. Steve Harris Says:

    @JRP The animation you mention is performed using Apple’s Core Animation, so its performance is somewhat determined by your Mac. Thanks for the feedback though.

    @zenrain – thanks also!