schussman.com logo

Lightroom 3 to iPad gallery workflow

(Or, an efficient way to use Lightroom to manage photos on your iPad)

I got myself an iPad a few weeks ago. Short review: It’s a big iPhone only in the sense that an iPhone is just a phone; that is, it’s hawt and I love it. (More ZOMG iPad articles to follow, I’m sure.)

The iPad really shows off photos, so I’ve been thinking about the best way to manage the photos that I want to put on it. With the non-beta release of Lightroom 3 this week, I think I’ve found a start at a workflow in the new publishing service that it offers.

First a note about syncing photos to the iPad. Like the iPod/iPhone, you have to select a single parent directory, and optionally subdirectories, of photos that you want to sync. The iPad stores photos in a series of albums simply named after the subdirectory where they live on your hard drive. In my case, I keep the photos that I publish out of Lightroom in one of a number of subfolders of a ~/Pictures/Exported Photos directory, but I don’t necessary want to sync all of them to the iPad. And, for the iPad, I wanted a little more control over the album naming for purposes of navigation — so I didn’t want to keep the directory names I’ve previously used for simple file management.

So: I have an “iPad” directory in my exports folder, and that’s what I sync to the iPad — along with all its subdirectories. I’ve copied a handful of existing albums into that directory, which isn’t a great solution but it does work for now. Apple has apparently decided that symbolic links cannot be synced [that would be a great capability, wouldn’t it?], so I can’t simply create links to the file locations of the existing, non-Lightroom albums that I want to sync.

iPad sync directory; Lightroom 3 publish service directories are subfolders of this

But the Lightroom albums are another story. Right now, I’ve simply created a couple of Hard Drive publish services, each of which publishes to a subdirectory of that “iPad” folder I described above. For example, the current flickr one simply sets up a set of export parameters for photos that will be published to a subdir.

Lightroom 3 Hard Drive publish service setup

But the slick part comes by making a smart collection that will publish everything I post to flickr in 2010. Simply build the smart collection within the publish service:

Creating a published smart folder in Lightroom 3

And then set it to include all photos that match some key criteria: Since I use Jeffery Friedl’s flickr plugin, I can call directly on the metadata it creates at export:

Published Smart folder settings to include a subset of Flickr-published photos

I now have a smart collection within a publish service, so the photos that I export that match that criteria will automatically be added. I’ll have to occasionally republish to the service itself; and each time I do so, the iPad will gather up those photos the next time I sync.

You can use the same methodology to set up hard drive publish services for any album you like; each service will appear on the iPad as an album, and if you create a smart collection (or a series of them) as part of the service, then they’ll be updated automatically as you perform you regular photo workflow.

One final note on why I think the publish service is an ideal tool for this workflow, rather than a solution such as simply exporting to a folder: The publish service will apply future changes to the service settings to all photos under its control. So, were I to decide to apply a watermark, a border (via, for example, LR/Mogrify 2), or adjust photo quality in order to save space on the iPad, those changes made to the publish service would result in the option to re-process the current photos:

Publish service allows for the updating of all published photos when changes are made to the service

In a sense, then, the photos produced via the publish service are truly synced — future changes will be managed and will trigger re-processing, if I choose. I think that’s pretty Good Stuff.

Another year of photo data!

Following in the modest two-year tradition I’ve established (see 2007 and 2008 posts), here is my 2009 photo data from my Lightroom catalog!

[ quick howto: Lightroom 1 & 2 (and 3) databases are in sqlite3 format, which means that freely-available tools can extract data from them. I use sqlite3, some shell scripting, and R (and occasionally excel) to produce summaries of that data. Why? Data offers some insight into the kinds of photos I take. Mostly, though, it’s fun. I’d be happy to expand on the actual code that goes into these plots, if there’s interest. ]

Below is a set of plots that summarize some of this year’s data. Click through to flickr to see the larger version.

2009 photo data!

What’s interesting this year? Well, crop ratios looked pretty similar to last year, so this year, for the first time (suggested in a post by Keitha, whose photos I admire tremendously, and whose Pentax lens set I envy with the fire of a million anti-glare-coated nine-aperture-bladed all-metal suns) I pulled out some information about aperture for each of the prime lenses that I shoot with. You can see these four frequency plots (for each of the Pentax DA 70mm F2.4 ltd, FA 50mm F1.4, FA 35mm F2.0 and DA 21mm F3.2 ltd lenses) in the left hand column of the image. Right off the bat you can see that I shot a lot with the FA 35mm this year (which is confirmed by the “overall lens use” plot on the right column). In fact, I took that lens along as my sole lens on a few long weekend trips to Ventura, CA, and the San Juan Islands, and really loved its performance. It does great at large apertures, but I also used it a lot for street shooting at f/8 and smaller apertures.

Runner-up in frequency this year is the FA 50mm F/1.4, which ordinarily I would say is my favorite lens (and it very much still is; it just wasn’t as convenient a focal length to take as my only lens on those vacations). Its sweet spot [where it’s sharpest but still has a nice narrow depth of field] is about F/4, which is where I primarily use it.

Neither the DA 70mm F/2.4 or the DA 21mm F/3.2 got as much use this year, but I really love some of the photos I took with those lenses. In fact, I carried these two lenses specifically for their light weight and trim size on the Flagstaff photowalk I organized in July.

Car / Cat Ranch house / wide Crow Pomegranite Backside Doorman

How did 2009 stack up to 2008? In terms of absolute frequency, nearly identical! I kept 1308 frames last year, compared to keeping 1340 in 2008. Far fewer of those are picks, or posted to flickr — though a good number are waiting for me to come back to, to finish workup or to make a print.

And that’s it for the 2009 photo stats! I did re-work my keyword network code, so perhaps can follow up this post with a little more about keyword relationships.

If you’d like to know more about extracting and summarizing info from your own Lightroom catalog, please let me know (and check out my other lightroom-related posts)

And, as last year, I hope soon to follow up with a report on my 2009 photo goals, and to set a few for 2010.

Lightroom: Build thumbnails and full-size exports simultaneously

It would be great if Lightroom could string export presets together, to perform operations sequentially or simply to perform several processes on one image to produce several different outputs.

The first use I’d put to this functionality would be generating thumbnails of exported images right along-side the full-sized image — a common task when producing images to upload to a blog, for example.

Absent that kind of flexible functionality, here’s a nice solution for generating thumbnails of images and full-size exports at the same time. Use Jeffrey Friedl’s Run Any Command plugin in conjunction with mogrify: Just insert “run any command” to an existing export preset, and then add the following as the Command to Run (assumes you have LR2/Mogrify installed, which of course you do):

cp '{FILE}' '{NAME}'-small.JPG ~/Library/Application\ Support/Adobe/Lightroom/Plugins/LR2Mogrify.lrplugin/LRMogrify.extras/mogrify -resize 50% '{NAME}'-thumbnail.JPG

The command makes a copy of your original image (which is available to “run any command” plugin via the {FILE} token), with “-small” appended to the filename, and then uses mogrify to resize the copy to 50%. You can optionally adjust the percent to scale or add a “-quality” option to the mogrify command.

You could perform this command more cleanly, without the copy step, if you have ImageMagick installed independently of LR2/Mogrify: The full IM package includes the “convert” command, which allows for output to a new filename. Since I didn’t want to separately install ImageMagick, I just make a copy of the file and then invoke mogrify via the path to the plugin.

Presto! Instant thumbnails, appropriately named, to accompany your export. Pretty good stuff, made possible by the great work of Jeffrey Friedl. Check out all his lightroom goodies.

What's in your Lightroom 2 catalog?

A question recently came up over in the Flickr Lightroom group about generating a list of all filenames or selected filenames in a catalog. There’s a pretty nice straightforward bit of SQL that will produce just such a list.

My new favorite tool for poking around in the Lightroom database is the SQLite Manager extension for Firefox. It plugs seamlessly into Firefox and is a very good tool—faster than the old sqlite3 browser, and with the added bonus of not requiring Rosetta to be installed under Snow Leopard. Plug the following code into the “Execute SQL” box (or kick it old school via the command line sqlite3 tool):

SELECT pathFromRoot, baseName, extension FROM AgLibraryFile JOIN AgLibraryFolder ON AgLibraryFile.folder = AgLibraryFolder.id_local

You can do even better by using the sqlite3 concatenate operate in the select statement:

SELECT pathFromRoot || baseName || '.' || extension FROM AgLibraryFile JOIN AgLibraryFolder ON AgLibraryFile.folder = AgLibraryFolder.id_local

That will produce a nicely-formatted list of files:

	2007-02-19/IMGP1609.PEF
	2007-02-19/IMGP1610.PEF
	2007-02-19/IMGP1611.PEF
	2007-02-19/IMGP1612.PEF
	2007-02-19/IMGP1613.PEF
	2007-02-19/IMGP1614.PEF

Still building, we can add in the absolutePath field to get a complete path — very useful for when you have files stored in multiple locations, such as offline storage or backup locations.

SELECT absolutePath || pathFromRoot || baseName || '.' || extension FROM AgLibraryFile JOIN AgLibraryFolder ON AgLibraryFile.folder = AgLibraryFolder.id_local JOIN AgLibraryRootFolder ON AgLibraryFolder.rootFolder = AgLibraryRootFolder.id_local

And, finally, because how often do you really need to see the paths for all of the files in your library, really?, we can limit the query to just files in a quick collection. This takes a couple of extra steps, because we can’t directly get to the keyword data from the path data:

SELECT absolutePath || pathFromRoot || baseName || '.' || extension FROM AgLibraryFile JOIN AgLibraryFolder ON AgLibraryFile.folder = AgLibraryFolder.id_local JOIN AgLibraryRootFolder ON AgLibraryFolder.rootFolder = AgLibraryRootFolder.id_local JOIN Adobe_images ON AgLibraryFile.id_local = Adobe_images.rootFile JOIN AgLibraryTagImage ON AgLibraryTagImage.image = Adobe_images.id_local WHERE tagKind like "AgQuickCollectionTagKind"

And there you have it. One more fun trick you can pull thanks to the sqlite3 database that underlies your Lightroom 2 catalog.

 

Check out the rest of my Lightroom posts for much, much more, including more database tinkering, keywording, and workflow.

Always more to learn about Lightroom

I’ve been reading the comments thread of this great post at Scott Kelby’s Photoshop Insider blog. It’s absolutely chock full of concise tips for getting more out of Lightroom, and it drives home to me that there’s always lots, lots more to learn.

Here’s my favorite new-to-me tip (Thanks, Scott!): Did you know Lightroom 2 has a ‘solo mode’? It’s a really fantastic way to eliminate scrolling up and down through the set of side panels. Ctrl-click beside a panel’s name and select ‘solo mode’ to turn it on, and then only the currently-selected panel will be expanded. Select any other panel’s name to auto-expand it while collapsing the prior panel. In conjunction with using cmd-1 through cmd-7 to toggle panels, it’s (to me) a nearly unbeatable way to navigate the panels with minimal mousing. (Note this works in both Library and Develop modes.)

Lightroom archiving and backup revisited

A while back I described my Lightroom 2 backup routine. A couple of recent blog posts around the web have given me some new things to think about with regard to backup and workflow.

Thomas Hawk writes about his workflow and Eric Scouten follows up with some thoughts. A core part of Thomas’s workflow is quick development of any and all keeper photos from a shoot and immediate export of those images to JPG. He re-synchronizes those JPGs with his Lightroom library and moves all the raw images offline (to a drobo; Jon at blurbomat extolls the drobo’s virtues, too) , and doesn’t apply keywords and such except to the finished JPGs.

Reacting to this, Eric notes a couple of things that get my attention (his emphasis):

  • The act of choosing what’s in and what’s out becomes an affirmative process rather than a process that’s about rejecting photos. Why not make the selection process a happy one?
  • This means the “selects” catalog is always in tip-top shape whenever I need to show someone my current work. Right now, the “main” catalog always contains some number of rough photos that haven’t been filtered out to the archive catalog.

By quickly moving from raw to selected and and “finished” JPG, the process builds the catalog of preferred images and gets the unused images out of the way. And applying keywords and other metadata only to the keepers dramatically reduces the overall metadata workload.

Where does this point me? I have a couple of thoughts:

  • Moving raw images offline, leaving only the JPGs readily-accessible, is a great way to impose some discipline over the “ooh, I might use that one someday” tendency. On the other hand, I’m not shooting in volume like Thomas is; while storage is an issue, there’s definitely a creative trade-off here. Until I shift my raw images to my network drive (usually about monthly), I’ll revisit a folder half a dozen times or more to see if there’s something interesting there that I didn’t see before. Thomas’s workflow, and that considered by Eric, are very different from mine though there is something appealing in it.
  • What this really opens up for me is a little bit of thinking about the intent and purpose of the photos that I take. Making sense of workflow and eventual archiving is a way to be conscientious of why I take photos. Is it because I’m an archivist? Is it for artistic purposes? Is it because I want to build a portfolio? I suppose I have a streak of all of those in me. I like the shooting itself, and I like the workup in Lightroom as well. So for me, it doesn’t quite make sense to workup photos quickly and move them offline.
  • However, it does make sense to develop a solid “keepers” collection as a sort of middle road; this collection gets updated frequently, but whether to keep those images as raw or JPG isn’t something I’ve quite sorted out.
  • What about those “someday” images? I need to consistently flag images with some potential and return to them regularly to re-evaluate.

Thoughts? How do you manage the possibility of future image development with your goals, limited creative time and storage space?

The year in Lightroom, by the numbers

I started last year to play with pulling data right out of my Lightroom catalog. How fun to combine interests in photography with my need to make data out of things. Last year about this time I posted some 2007 photo stats, and with the release of Lightroom 2 I came up with some keyword network maps of my flickr images.

Over at The Online Photographer, Marc Rochkind did some writing about meta metadata and released a tool for OS X that produces much more summary information than I had previously considered: His tool produces by-lens statistics on cropping and aspect ratio in addition to focal length usage. This generated some thoughtful conversation about composing in the viewfinder versus cropping, and Marc’s work spurred me to think more about my own stats, and so I went back to my own Lightroom 2 catalog with the sqlite browser and R to see if I could reproduce for myself some of the more interesting data that Marc’s tool generated. After some tinkering, I think I have a functional, reusable set of R tools for generalized reporting of Lightroom image data.

Like Marc’s ImageReporter, I can filter by image type, picks, ratings, Quick Collection, camera model (though this matters less for me since I have one P&S and one DSLR) and time period, and I added filtering by color label as well — hey, just for fun, even though I don’t use the color labels (I generally get rating fatigue using anything more than picks.)

So, what do I have? First, a reproduction of the stats I checked out last year: Monthly photos and focal length:

The year in Lightroom

I continue to primarily use my prime lenses, and my picture-taking appears to have notched down dramatically as compared to 2007. This is partly because of work, of course, but also because I’ve become much more selective about what I actually keep in the catalog.

We can break out focal length a bit more. For the two zooms that I use on my K100D, what are the mean focal lengths?

> lensFL [1] 5.8-23.2 mm 15 [3] 85.0 mm f/1.8 85 [5] smc PENTAX-DA 18-55mm F3.5-5.6 AL 31 [7] smc PENTAX-DA 21mm F3.2 AL Limited 21 [9] smc PENTAX-DA 50-200mm F4-5.6 ED 121 [11] smc PENTAX-DA 70mm F2.4 Limited 70 [13] smc PENTAX-FA 35mm F2 AL 35 [15] smc PENTAX-FA 50mm F1.4 50

So that’s kind of interesting, suggesting that I use the 200mm zoom at about the middle of its range. But the mean isn’t necessarily informative. Here’s a plot of focal length for one of those zooms:

Focal lengths plot, DA 50-200mm lens, 2008

So, I use the 50-200mm lens primarily for shots at either extreme of its length, and I already have a 50mm fixed lens that takes better photos than the zoom at that distance. Moreover, breaking out just picks with this lens shows a three-to-one preference for 200mm than for 50mm. I think that means I need a long prime. Ka-ching!

I can also consider crop: How am I doing at composing in-camera? Here’s how often I crop, by lens, as a percentage:

	smc PENTAX-DA 18-55mm F3.5-5.6 AL   9.13 %
	smc PENTAX-DA 21mm F3.2 AL Limited 17.67 %
	smc PENTAX-DA 50-200mm F4-5.6 ED    6.93 %
	smc PENTAX-DA 70mm F2.4 Limited    23.78 %
	smc PENTAX-FA 35mm F2 AL           10.71 %
	smc PENTAX-FA 50mm F1.4            24.67 %

And, when I do crop, how much of the original composition do I keep?

	smc PENTAX-DA 18-55mm F3.5-5.6 AL  78.3 %                            
	smc PENTAX-DA 21mm F3.2 AL Limited 81.8 %                            
	smc PENTAX-DA 50-200mm F4-5.6 ED   81.6 %                            
	smc PENTAX-DA 70mm F2.4 Limited    80.9 %                            
	smc PENTAX-FA 35mm F2 AL           83.4 %                            
	smc PENTAX-FA 50mm F1.4            82.5 %

So, I’m cropping quite a bit. As Marc found in his exploration, these numbers go up when I filter by picks. I was surprised that I crop as much as I do with the DA21mm in particular, since I think of my use of it as being mostly for wide landscapes; but even those often enough are a bit crooked, enough to warrant at least some adjustment of tilt —- and Lightroom calls that adjustment a crop (fairly).

Does cropping mean I do a poor job at composing in-camera? Possibly. I have to admit that knowing I can crop gives me a conscientious freedom when I’m shooting, but these numbers give me something to think about. Maybe careful composition will be something to work on as I go forward.

We can cut all this in a few other ways. I’d like to take a look at my common keywords during a given time period, for example, but that will wait for the follow-up post, I think. This is more than enough nerdery for one January 1st afternoon.

Quick Lightroom 2 Tip: Show untagged images

I haven’t used Lightroom 2’s library filter extensively, but it’s really pretty powerful and I found a good use for it this morning. Having tagged a big batch of new images somewhat helter-skelterly, I found myself wondering if I could view just the images that I had not yet tagged, so that I could take care of them and be all done with this round of keywording. The library filter does this handily:

lightroom 2 metadata browser - setting presets

Open the filter pane (hit \ to activate and deactivate it), then select the “text” controller, and set it to “Keywords” “Are Empty”, and there you go: Lightroom 2 will display just the untagged images in the current view. Very handy, and the display will update as you proceed to tag, slipping newly-tagged images out of view.

I saved this library filter view as a preset using the dropdown controller available in the upper right of the pane, so I can quickly retrieve it any time I like.

Update: I see the following in a good post over at the O’Reilly Lightroom blog:

One of the example smart collections that is installed with Lightroom is Without Keywords. Take a look at the image count for this collection. If it’s anything other than 0 then get in there and add some keywords!

Nice! I haven’t much explored smart collections. That built-in collection is a great find.

From Raw to Rawwwr: Lightroom 2 Raw Development for the Rest of Us

I have a casual obsession with Lightroom 2 and have had a load of fun tinkering with the Lightroom catalog database to extract information about my photographs. More recently, I’ve been writing over the past handful of weeks about my workflow in Lightroom 2 from the perspective of an enthusiastic but non-professional user. I’ve previously covered organization and keywords. This time around, the topic is using LR2 to work up raw photos.

Why would one shoot in raw? Raw is an image format meant to preserve as much original image data as possible; that information can be manipulated much more flexibly and effectively than the comparably limited range of image data contained in a JPG or TIFF image. A raw image with bad exposure or white balance can be salvaged, while a JPG — lacking the image information of raw — offers far less latitude for recovery. Meanwhile, JPGs produced by most digital cameras have some amount of sharpening, exposure, and contrast adjustment applied in-camera, while raw images are essentially straight representations of what the camera “saw” when taking the photo. This means that raw images will almost always need a bit of color tweaking and sharpening. Hence the eternal quest for a reliable, easy-to-apply workflow for “developing” raw images to best effect, and the subject that brings me to the MacBook today.

After importing and doing preliminary tagging, flagging and deleting of images, I start in on working up the raw images.

White Balance

White Balance is the first thing to tackle. The point of correcting white balance is to give the right color tone to the “neutral” (white or grey) colors in your image. In other words, you want white to look white. If the WB is off, your photos will have an unnaturally cool or warm cast.

In Lightroom 2, you can adjust white balance in both develop and library modules using presets for different light types, and in the develop module via the temperature sliders. (There are a couple of quick-develop buttons for temperature, but I generally avoid using them.)

The most common white balance adjustment I make starts with selecting the “tungsten” preset. This immediately removes the too-hot feel of photos taken in incandescent light — ie, most sources of indoor, non-flash light.

LR2 screenshot

After using the Tungsten preset, it’s usually necessary to make a few adjustments: This is when I use the Tint and Temperature sliders in the develop module for subtle shifts, often to add just a bit more warmth to the light by nudging the Temp control a tiny bit back to the right. It’s a good idea to swing the Temp control around a bit just to see what kind of difference it makes to give a bit more warmth or coolness to an image.

Mixed light provides a dilemma for white balance. Light from an overhead lamp plus natural sunlight through a window, for example, can be tricky to adjust. I find that setting the Tungsten preset will cool off the overall color cast, but will also amplify the parts of the image that are already a bit cooler — the natural sunlight — giving them a sharp blue tone, especially around highlights. My solution to this is to desaturate the blues a bit, which takes the edge off that coolness. In Lightroom 2, you have even more ability to make that correction by using the adjustment brush to spot-desaturate the highlights that are made too cold by adjusting to Tungsten. The below shot illustrates an image that’s been tweaked by first presetting WB to Tungsten, and then backing off the blue saturation just a few points.

LR2 screenshot

But wait, there’s more! While the Tungsten preset is good, it’s not always quite right. As alternative, use the White Balance Dropper — that’s the dropper button next to the WB selector in the develop module. Click it, then click a spot in your image that should be nice and neutral, a white or grey spot. Presto! Instant white balance. When I can’t quite decide on the right balance, I often use this tool on various spots of the image to quickly try out a range of white balances. The shortcut for the WB dropper is W, which you can use in both grid and develop modes.

Once you’re happy with the white balance of a given image, you can quickly apply those settings to the remainder of similar images with the “sync settings” command: Swap over to grid mode, select the corrected image first, and then click-select the others with similar light, and finally sync ( shift-cmd-S).

There’s much more to the raw workflow: Exposure, sharpening, and so on, and I’ll tackle some of those next time.

My Lightroom 2 Backup Strategy

Related: A more recent post about archive and backup in Lightroom.

With this morning’s comment from martie asking about a crashed hard drive, I got to thinking about making my own Lightroom 2 backup plan a bit more automated and reliable. My general approach is to periodically copy my catalog file and image directories to an external hard drive, but there’s been nothing systematic about it until now.

I’ve previously described a bit of my Lightroom file structure, noting that I import new photos into a single directory per import. As part of a strategy to save space on the MacBook where I do my actual work, I periodically move those folders to an external hard disk currently named Grundle. This is simply a matter of dragging the folder, in the left-hand directories pane of Lightroom, from one hard drive to another.

Lightroom display of multiple drives

While this copying step is manual, the rest of the system is now automated, thanks to this tutorial at MacResearch and a bash script by Aidan Clark. The bash script took just a bit of tinkering to work with Lightroom’s catalog file, which by default will have rsync-breaking spaces in it, and to perform the second backup from the external volume to the iMac. I’ll post those specific and very minor modifications if there is interest.

Here’s the final result: Using OS X’s launchd tool, whenever I mount Grundle on my MacBook, whether via network or direct firewire connection, my Lightroom 2 catalog file is copied to Grundle using rsync. And, whenever I mount Grundle on the upstairs iMac, a similar combination of launchd and rsync copies both the catalog file and the image directories from Grundle to the iMac. This means that in the course of regular use of my two Macs and that external drive, both my Lightroom catalog and folders full of images get backed up.

One caveat to this system is that the backup of the image folders still involves that manual step of moving them from the laptop to Grundle. I could automate this the same way the catalog backup is done, but that could potentially mean trying to backup a gig or more at a time over the wifi network — a time- and bandwidth- consuming process that isn’t really necessary. Now, the obvious down-side is that the newest photos I’ve taken are always the ones most vulnerable to data loss, and that’s obviously not a highly desirable thing. But I’m satisfied with my current workflow of moving folders to Grundle generally when I’m done working with that set of images. I’ll continue to think about this situation and may come up with some additional redundancy for that stage of processing.

Update: Okay, I buckled. A bit more tinkering and I now have my current folders of raw images copied to Grundle. After I relocate the folder using Lightroom, the folder will disappear from the backup directory, so I don’t have redundant backup files stacking up anywhere. Nice and clean, and everything’s safe.

Update the second: One item I neglected to mention in the original post was the automated backup feature built in to Lightroom: Available in the catalog settings menu (alt-cmd-,) this feature performs scheduled backups of your catalog file only, to a location you specify, and can be set to run a backup on any of several schedules. My process above includes allowing that backup to run weekly — it never hurts to have a little more failsafe security. The benefit of automating getting that backup to another hard drive is one more important layer of keeping safe your data.

Dealing with duplicate keywords in Lightroom 2

I’ve noted recently how slick it is to be able to use nested keywords in Lightroom: It’s a piece of cake to select a set of photos, hit cmd-K, and enter “mount humphries > mountains” to assign the “mount humphries” as a child keyword of “mountains”.

However, as noted by a poster to a thread about keywording over on the Flickr Lightroom group, this creates a potential problem: If the child keyword is already used, Lightroom 2 will end up creating a duplicate keyword; you’ll end up with one “mount humphries” without a parent, and one “mount humphries” keyword with the parent of “mountains.” So, by trying to be hierarchical with your keywording, you’ve actually splintered your keywords. Not helpful!

Having already found a fun way to explore relationships between and frequencies of my keywords, it occurred to me that I might have some ready-made tools to help with this situation: The need to find and deal with duplicate keywords.

As this article became more popular, I worked through a couple of alternative methods and organized things a bit further. To date, I describe three methods of identifying duplicate keywords:

  • Full auto: Requires some scripting but is the most expedient way to go about it (and my favorite).
  • Semi-auto: Requires the awk tool to identify duplicates but doesn’t rely on any sqlite3 code to pull from the LR database.
  • Full manual: Uses LR’s built-in export tool and MS Excel to get what you want. Lots of steps, but it works.

After finding your preferred method, read on to see what to do with all the duplicates you identify.

The full auto/scripted method

Thanks to Dieter for putting me on to a much quicker, straightforward way to identify duplicates using SQLite only — no awk:

select count(name) as num, name from AgLibraryKeyword group by name having num > 1

I’ve kept the original SQLite+awk method below for posterity:

First I needed to find any duplicates created — potentially — by the ad-hoc nesting of keywords. A quick modification to my keyword frequency script produces a list of dupes:

# display a list of duplicate keywords in Lightroom 2 cp ~/Pictures/Lightroom/Lightroom\ 2\ Catalog.lrcat ~/lightroom.lrdb /usr/local/bin/sqlite3 -csv ~/lightroom.lrdb 'select ImageCountCache, name from AgLibraryKeyword;' > /Users/alan/lr-keywords.csv awk -F , '{print $2}' lr-keywords.csv | sort -n | uniq -d rm ~/lightroom.lrdb rm ~/lr-keywords.csv

As with the frequencies, this short script makes a backup copy of my Lightroom 2 database, then calls on sqlite3 to extract the list of keywords; the difference is in the 4th line, where I use awk to pull out the tags and then pipe them through two built-in unix functions to print a list of any duplicates. In my case, it yielded the following:

Running the duplicate keyword script

Perfect! A list of keywords that appear to be duplicates.

Note: The script above works with OS X Tiger and requires an upgrade to the default version sqlite. It ought to work with Leopard as long as sqlite is present. Windows? Don’t know; either via cygwin or separate binaries, awk and uniq should be available for Windows, and there is a sqlite for Windows download at the above link.

Skipping the sqlite step

Update 17/Dec/2008: If you’re not eager to delve into sqlite, you can still make this work, but you’ll still need to have the awk tool. OS X users, you should be good to go, since awk comes with the OS; Windows users, you can download awk for Windows. First, manually export your keywords list from your catalog: Metadata > Export Keywords…, and save the file as lrkeywords.txt, and then run the following one-liner script from a shell/terminal:

awk 'BEGIN {FS = " "}; {$2 = $2; if (match($0, /{/)0) print}’ lrkeywords.txt | sort -n | uniq -d

Just as in the sqlite version, this one-liner parses your keywords file and returns the list of keywords that appear more than once and are not identified as synonyms. You can then reconcile the duplicates as described above. I prefer the single-step version that extracts directly from the database, but hope this is useful to a few folks.

The Full Manual Process

If you’re averse to both the sql steps and to using awk, you can use MS Excel to identify the duplicates. I think it’s far more cumbersome than either of the above processes, but it works. Here goes:

Manually export your keywords from Lightroom: Metadata > Export Keywords ….

Open that export file in Excel: Open as a text delimited file, but uncheck all the delimiters; you don’t want excel to parse along spaces or tabs, since both of those characters appear in the file but not as record separators.

You should see a single column of keywords something like this:

screenshot

We need to slightly clean up that column to remove leading whitespace. We’ll use the CLEAN function to do that: In the column next to your keywords column, enter the formula =clean(a1), and then drag that formula all the way down the keywords column.

screenshot

In col B, you now have a whitespace-trimmed set of keywords, but because of the clean() formula, you can’t manipulate it further. Select that entire column, copy it, and then use paste special to paste the column as values into column C.

screenshot - cleaning up keywords list with paste special

Now we’re ready for the final steps: Sort, flag, and filter. Select column C, then go to Data > Sort, and sort ascending by column C. Now, in column D, drag the following formula from the first row to the last: =IF(C2=C1, "!", "").

screenshot - applying the duplicate flag formula to the sorted list

That just fills the column with a flag if column C has a duplicate. It’s a low-budget search, but it works as long as the list of keywords is sorted. Finally, use Data > Filter > Autofilter, and click-select the ! in column D. You’ll now have a filtered list of duplicates from your original keyword list, which you can resolve as described above. Note that you’ll have a number of keywords surrounded by {} or [] brackets; these are keywords entered as synonyms or categories, and you should ignore them when you are addressing duplicates.

screenshot - the list of filtered duplicate keywords

You can see from my current duplicate list that I’ve been working heavily on food-related keywords as we cruise through the holiday season.

All told, that whole manual process should just take a couple of minutes once you have the steps sorted out. Because the steps are manual, it’s not as easily-repeatable as the automated sqlite+awk approach, but it does work. I hope someone finds it useful!

Dealing with the duplicates

Whatever method you’ve employed, at this point you have a list now — let’s check out if it means what I think it means. Switching back over to Lightroom, I can filter for all photos with the “mount humphries” keyword:

Lightroom listing duplicate keywords

Sure enough, I have 12 images tagged with “mount humphries”, and 11 images with the same tag set as a child of “mountains” (as an aside, I see that I have well over a hundred images with the “mountains” tag that could probably use some more granular tagging).

My first impulse was to try to just drag the non-child “mount humphries” into the “mountains” tag; this works, after all, with other keywords. But in this case, it won’t do the trick, presumably because there is already a “mount humphries” keyword there — Lightroom won’t let me add a same-named child.

To reassign the keyword to the parent, you need to take a few more steps: First, click the right-pointed arrow to the right side of the duplicate, non-child keyword; this will navigate to all images assigned that keyword. Then, in the grid view, simply select all (cmd-A), and then check the child keyword to add it to all of the selected images (the checkbox is to the left of the keyword, and appears when the mouse cursor hovers over the keyword), and you’ll see the count increase accordingly. Next, un-check the duplicate, non-child keyword in the keywords panel. You’ll see its count drop to 0. The order of those two check-uncheck steps is important: If you uncheck the non-child keyword first, you’ll end up with an empty selection and nothing to apply the proper keyword to.

Lightroom displaying duplicate keywords unchecked

There! My “mount humphries” keyword as a child of “mountains” is now assigned to all 23 original images, and I can delete the duplicate, non-child tag.

So, with an approach like this, ad-hoc keyword nesting shouldn’t be feared: We can identify duplicates created by nesting, and, in a matter of seconds, apply the same nesting to any previously-tagged images. And, once you’ve resolved the duplicate, any further assignment of the focal keyword will always assign it, appropriately, as a nested tag. Pretty slick, I do say.

A Lightroom 2 workflow for the rest of us

(Welcome to readers coming via the nice link from the Lightroom team at Facebook! Consider checking out the rest of my Lightroom writing.)

—-

While Lightroom gets a lot of love from professionals who take a thousand photos at an event and need to manage a paying production workflow, it’s also great for enthusiast-level users like me: I usually work on the scale of a weekend’s or a trip’s worth of photos that need organization, a place to live, some pruning and keywords, and whatever sorts of RAW workup that’s necessary. Last week I wrote up a few general Lightroom tips with some vague intention of documenting a practical workflow; here are two high-level elements of my workflow: Import/File structure and metadata/keywords.

Import and file structure: I shoot in RAW and use Lightroom to import photos directly from the SD card via USB card-reader. Lightroom presents a number of options for storing those imported files: In the past I put imported photos in by-day folders, meaning one directory for each file date. So, for a handful of days’ worth of photos I’d have a hierarchy of directories like 2008-09-01, 2008-09-02, and 2008-09-03. I’ve recently switched to a storage option that, so far, I really prefer: I import photos to a single directory based on import date rather than file date, with a descriptive suffix to that directory. The nice upside to this approach is that a weeks’ worth of photos — a five-day trip somewhere, for example — all end up in a single directory, and can all be viewed and processed as a set (without having to put them all into a collection). Since Lightroom is so handy with metadata, I don’t lose anything by stepping up in granularity from by-day storage to by-import storage. Additionally, having fewer directories hanging around makes it easier both to backup and to browse my photo file tree with other applications.

Below is a screenshot to illustrate the import and directory naming stage:

Lightroom Import Dialog

On import, I assign a metadata preset and whatever keywords are applicable to the entire import set of photos.

After import is a first pass to identify picks and rejects and assign keywords, which I described last time around: Set grid mode to a large-ish preview size, and use shift-P and shift-X in grid mode to mark any clear favorites and rejects. After that pass, it’s keywording/metadata time.

  • Metadata: Lightroom has two-dozen or so non-keyword metadata fields that, frankly, I don’t have much use for. I tend to use only a few of them (in the “Image” metadata category) to assign geographic location of photos taken: I have metadata presets for common locations, and I assign the rest on an ad-hoc basis. At export time, some of this metadata is included as keywords, so my exported photos end up with, at least, city and state information attached.
  • Keywords: You can keyword forever in Lightroom. I start keywording with a single keyword set that broadly covers my most common uses (The “edit keyword set” function is accessed in the Keyword Set menu in the right-hand Keywording panel): I call this one “Common/Home”, and it contains, among other keywords, “Heather”, “dogs”, “flickr”, and “food” tags. I know the order of these tags and can quickly make assignments with alt-num combinations. With my “Common/Home” set active, alt-7 tags a photo with flickr, alt-8 with Heather, for example. Wondering what your common keywords are? Try exploring your database for common lightroom keywords, and take advantage of nested keywords and keyword hierarchies for the greatest efficiency in assigning keywords.
  • In Lightroom 2, the related keywords function adds a whole new dimension to commonly-used tags. Once you’ve assigned one or two from a preset menu (as above), you’ll get a set of keyword suggestions, as well. Lightroom 2 really does facilitate both identifying and applying keywords in a useful way.
  • On applying keywords: As with metadata, I’ve found it most useful to reduce the size of previews in the grid window and apply keywords in groups to a selected set of images first (as many as possible; ctrl-click (shift-click for a range) or shift-arrow with keyboard to select multiples), and then step through photos one-by-one, adding additional keywords as needed.
  • Lightroom lets you display keywords for a given image in a number of ways. I like to keep the keyword display set to “will export” — That’s generally the most complete display of the tags assigned to a given image, including synonyms and parents.

Possible items for next time? Super-basic RAW development, exporting, and collections: See my intro to RAW development!

Photography: Hitchhiker's Guide to Lightroom

I spend a lot of time in Lightroom 2 these days. I’m nobody’s pro, but I shoot a lot of photos, and after having used Lightroom (and now Lightroom 2) for a while now, I think I have a pretty good, simple, enthusiast-style workflow sorted out. I’ll summarize the workflow itself (importing through working up images) in follow-up post. Here are a few general tips that seem to work well for me:

Essentials, or Stuff I use constantly: I use Picks and keywords extensively. Reviewing newly-imported photos, I mark anything that I like right off the bat as a Pick by simply hitting shift-P as I scan through the gallery (and shift-X to immediately mark others as rejects; the shift modifier will mark the current photo and move on to the next shot). As I revisit a set of shots later, I find myself repeating this process; while those subsequent passes primarily identify further Rejects, I do occasionally find more Picks after starting to work up other photos. After each pass through a gallery, I use cmd-DELETE to remove (and delete) all the Rejects.

This has been a nice insight for my process: It means that I am fairly conservative when it comes to Rejects. That is, I don’t mark as Rejected 1) unless a photo is obviously bad (bad focus, blur, composition I really dislike, etc.) OR 2) until I’ve spent some time on photos in a set that I do like right from the get-go. This frequently helps give me a sense for appealing qualities of photos that I might not have noticed or thought of initially.

With a gallery through at least a first pass of identifying Picks and Rejects, I apply keywords. As with many aspects of processing photos, Lightroom has lots of ways to do this. There’s a jobber called Keyword Painting that I don’t use, because it’s always been much faster to simply select sets of photos and then apply keywords to the selection. In Lightroom 2, cmd-K focuses on the keyword entry box, which will auto-complete as you type. Lightroom 2 also has “recommended keywords” functionality, so that as keywords are assigned to a photo or set of photos, a new set of co-occurring keywords is identified and displayed for easy additional assignment.

Although I like to use a large-ish image preview (hit = to increase the size of preview images in the gallery grid) for screening for Picks and Rejects, for keywords I like to shrink the grid size (keyboard shortcut -). This fits more images into the grid and allows me to select larger sets for group assignment of keywords.

Lightroom allows for keywords to be nested, and there’s a great shortcut for accomplishing this: When entering keywords, separate child from parent keywords with a > sign: flickers > birds, for example, or burgers > food.

Simplifying, or Things I don’t use in Lightroom: Beyond keywording, Lightroom has at least a trio of way to identify and categorize photos: You can flag photos as Picks, label them with colors, and rate them with zero through five stars. I don’t use colors or stars at all. They may be highly useful for some situations, but they just clutter the cognitive space where I think about my photos: “Is this a three-star green photo, or a four-star blue one?” So except in the rarest circumstances, I haven’t yet found a use for ratings and color labels.

Indispensable keyboard shortcuts: There are grundles of these, but the shortcuts I use all the time are:

  • G, E, D: Gallery, Editor, and Develop modes
  • P: Mark as Pick (modify with shift)
  • X: Mark as Reject (modify with shift)
  • cmd-delete: Remove Rejects (optionally delete from disk)
  • cmd-K: Assign keywords
  • W: Jump to White Balance selector in Develop mode
  • R: Crop tool in Develop mode
  • J: Show clipped darks and highlights (Developer only; in gallery, changes display of thumbnails)
  • L: Cycle the lights (view on black)
  • tab/shift-tab: reveal/hide menu panels

Next time: The library filter, file organization, workflow, and Lightroom+Flickr?

Frames and borders with Lightroom 2

Update 7/26/2009: There are some fantastic new features of LR2/Mogrify to check out:

  • Relative-sized borders (which I’ve written about before) are super.
  • As is relative-sized annotation text. No more calculating your scaled text size to apply to an export, since you can set the size to a percentage of image width or height. Very cool. (This goes for watermarks, too)
  • More border features! A checkbox for “identical borders” makes it much easier to set uniform borders. And inner borders are now supported for the creation of an inset frame with variable opacity.
  • Setting compression by file size: Specify a file size and LR2/Mogrify will compress the image accordingly so as not to exceed that file size.

Give these a try and don’t forget to donate to Timothy’s work if you find it of use.

Also, note that you can use Lightroom’s post-crop vignette feature to generate curved borders (in black or white) with less flexibility but also without plugins.

Update 12/21/2008: Much of this writeup now has more historical than practical value, since Timothy Armes has updated LR2/Mogrify to support multiple border options within the plugin’s own control panel. You can specify different-sized frames & borders without any extra monkeying around. Nice work, Tim!

A question recently came up in one of the Lightroom groups over on flickr about creating images with large borders on just one side — space within a frame to place a title, for example, but just along a single edge of an image. The poster wanted to create images such as those found here, and wondered if it was doable without diving out to an external tool like Photoshop. The first working proposal was to use a graphical frame applied in the print module, but that isn’t an ideal solution for me; it still requires setting up that frame with something like Photoshop, and to apply it you have to switch modes. So I tinkered a while with a photo I took a couple of nights ago, and managed to get what I think is a nice solution via a direct export from Lightroom using the fantastic LR2/Mogrify plugin from Timothy Armes.

The out-of-the-box options for this plugin don’t provide a capability to create different-sized borders to an image, but the underlying engine for the plugin, Imagemagick’s mogrify tool, does — after a fashion. So, in a nutshell, the trick is to use the command line element of LR2/Mogrify in addition to its other features, to add to the picture’s canvas size before performing the other operations.

Monte Vista neon

There are just a couple of tricks to get this to work smoothly using LR2/Mogrify. First is to add the extent command to the mogrify configuation, specifying the resulting size of the image you want to export:

I’ve specified the command -background white -extent 3008×2158 to be prepended to the mogrify command line that LR2/Mogrify will execute for me. I’m exporting an original image that’s 3008×2008, so I’ve specified 2008×2158 to the final image — adding 150 pixels, which will be filled with a white background. Next I use the built-in features of LR2/Mogrify to add the colored frames and the text overlay.

Because the extent command was applied to the beginning of the command line, the borders will be applied to the new image — the one with the bigger lower border created by extent.

The text overlay is just a bit strange. Note that instead of specifying the text to fall at the bottom, I’ve placed it at the top center of the image, with an offset of 2158 pixels. For some reason, directly placing it at the bottom center reverses the position of the new white border — it ends up at the top of the image, through some kink of mogrify that I can’t quite sort. It’s easy enough to compensate with the offset.

Export away, and that’s all it takes. You’ve built an image with a nice broad frame and caption, all right from Lightroom’s export panel. No Photoshop or print module necessary. Fun.

Lightroom 2: Related Keywords are Dreamy

As it happens, Lightroom 2.0 has just the thing I daydreamed about a handful of months ago. The new version’s data includes a table of keyword co-occurrences that makes it possible to produce things like this:

My flickr tag neighborhood

This graph shows keyword relationships that occur within a hop from my “flickr” keyword — which I use to keep track of photos that I upload there. In other words, it’s sort of a descriptive keyword neighborhood of what I’ve put up on flickr.

Color is a little subjective. The darker blue, the higher the ratio between unique neighbors and total neighbors. That is, darker blue nodes are connected to relatively few unique other neighbors than the lighter blue nodes.

Of course, you could use any focal keyword for this kind of thing: Starting with a lens-specific keyword would produce a rough map of the neighborhood associated with that lens, and might reveal how I tend to use that lens. The possibilities are pretty endless — and totally a fun kick in the pants to tinker with.


About, the short version

I’m a sociologist-errant. This site is powered by Textpattern, Pair Networks and the sociological imagination. For more about me and this site, see the long version.

RSS feed