Tag Archives: mac os x

find hidden files with lsflags, show them with chflags

I want to go fishing in my work Mac’s ~/Library folder for some app settings files, but Apple has decided to hide it from me. I could use -shift-G, or open ~/Library, but that’s not the kind of guy I am. Why is it hidden? The filename doesn’t start with a dot, so the Finder’s going on something other than Unix convention there. Were this 1998, I’d launch ResEdit and clear the folder’s HFS hidden flag. Maybe that’s still around in some form? I’ll look at the file’s extended attributes, because I’ve seen other HFS stuff like resource forks in there.

Arachnoscope:Desktop steelpangolin$ xattr -l ~/Library
00000000  00 00 00 00 00 00 00 00 40 00 00 00 00 00 00 00  |........@.......|
00000010  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|

Well, I was hoping for something more like com.apple.Hidden, but this is a start. Except it’s completely opaque. Googling… aha. A hidden flag is mentioned in chflags. But how do I confirm that this flag is actually why the Library’s omitted from the Finder’s listings? Is there any way to read flags?

Arachnoscope:Desktop steelpangolin$ ls /usr/bin | grep flag

Nope. Back to the man pages; I’ll have to write one myself: lsflags.c.

Now I’ll give it a go:

Arachnoscope:Desktop steelpangolin$ gcc -std=c99 -Wall lsflags.c -o lsflags
Arachnoscope:Desktop steelpangolin$ ./lsflags ~/Library
/Users/steelpangolin/Library: hidden
Arachnoscope:Desktop steelpangolin$ chflags nohidden ~/Library
Arachnoscope:Desktop steelpangolin$ ./lsflags ~/Library
Arachnoscope:Desktop steelpangolin$ xattr -l ~/Library

Hooray. The flag shows up, and it is indeed stored in the com.apple.FinderInfo xattr, because both disappear when I remove it. But what’s this, at the end of the first man page I found?

You can use “ls -lO” to see the flags of existing files.

Note to self: read all the way to the end next time.

Tagged ,

hello world, meet gpt-surgeon

gpt-surgeon has a new home on Launchpad! Look for it at https://launchpad.net/gpt-surgeon. Downloads compliant with the Launchpad release scheme (GPG-signed tarball) available here: https://launchpad.net/gpt-surgeon/+download

I’ve decided to go with GPLv2 as the licensing scheme, no copyright assignment required, to keep this simple utility open and available to the public with a minimum of hassle and a maximum of fresh code.

gpt-surgeon will still be available here on Bat Country for a time, but fresh development and new versions will be on the Launchpad site. This version will be frozen as soon as the Launchpad version is available.

Tagged , , , , , , ,

more on: invalid BS_jmpBoot in boot block: 000000

I’ve been getting a lot of comments that say “invalid BS_jmpBoot in boot block: 000000” happens with  Apple’s Boot Camp HFS drivers from Snow Leopard as well as MacDrive. The act of assigning a drive letter to the partition seems to cause the GPT corruption. This would seem to make it a Windows issue. I do not speak for my employer in any official capacity, but when my current high-priority tasks are done, I’ll ask around at work and see if anybody knows anything about this issue.

Meanwhile, keep sending in your experience reports! The more data, the better.


update: Comments have been turned off for this post; all questions, comments, suggestions, and support requests about gpt-surgeon must be submitted through the gpt-surgeon LaunchPad site and should be accompanied with disktype reports.

Tagged , , , ,

Unicorn Support Would Be Easier

Unicode has two ways to represent composite characters like accented Roman letters. If you want to write a ü (that should render as a lowercase u with umlaut/diaeresis/two dots over it, if your browser is any good), you can represent it as the single Unicode codepoint LATIN SMALL LETTER U WITH DIAERESIS (U+00FC), or you can represent it as LATIN SMALL LETTER U (U+0075) followed by COMBINING DIAERESIS (U+0308). Since these are, in fact, the same letter, Unicode defines normalization forms for things like comparing and sorting strings. In the NFC form, all composite characters that have a single codepoint representation are represented as that single codepoint; in the NFD form, they are all decomposed into multiple codepoint representations. If you pick a normalization form and stick with it, then you can ensure that ü is always equal to ü no matter which representation it came in with.

If you don’t do normalization, then annoying things happen. The setup is as follows: I’ve got a server machine running Debian 5 Linux, serving files from an XFS filesystem through Samba, talking to a client Mac running OS X 10.5 (Leopard) and mounting the share using mount_smbfs(8). Some files and folders have accented letters in their names. Now, they display fine in Finder directory listings. Problem is, when you click on them, they disappear!

It took some poking around in Wireshark to figure out what was going on. I noticed that the file names being received from the server in directory listings were not the same as the ones that were later being requested: specifically, characters like our friend ü were decomposed in the listings, but composed in the requests. Samba would then report the requested files as missing.

I’m honestly not sure who to blame here. It could be OS X’s fault for not keeping track of received filenames somewhere so it could send out the same ones it got. It could be Samba’s fault for not normalizing incoming requests before looking up files when it’s set to use Unicode. It could even be XFS’s fault: XFS filenames are byte strings and can include any bytes other than the ASCII codes for / and NUL. The Mac OS filesystem HFS+ uses a normalization form that is (almost) NFD, so filenames are stored on disk in one format only, and other representations are illegal. XFS does not appear to do any processing of filenames. It’s actually possible to create two files named “ü” in the same directory, or at least two filenames that display as “ü” when interpreted as UTF-8 byte strings, decoded to Unicode code points, and displayed by something that can handle Unicode characters.

Fortunately, there is a workaround. If Unicode filenames are converted to NFC on disk on the server, they appear to survive the round trip to the Finder and back just fine. Python’s unicodedata module has a normalize() function that makes this pretty painless.


update: here’s some working sample code to NFC-ize everything in a folder hierarchy.

import os
import unicodedata

for root, dirs, files in os.walk(u'/path/to/base', topdown=False):
    for entry in files:
        nfc = unicodedata.normalize('NFC', entry)
        if entry != nfc:
                os.path.join(root, entry),
                os.path.join(root, nfc))
            print os.path.join(root, nfc)
    rootparent, rootentry = os.path.split(root)
    nfc = unicodedata.normalize('NFC', rootentry)
    if rootentry != nfc:
        os.rename(root, os.path.join(rootparent, nfc))
        print os.path.join(rootparent, nfc)
Tagged , , , ,

the new spaghetti code

Found Apple’s Quartz Composer last night. Proceeded to write a clock. It’s very easy to use, but it could be much better if it had some sort of layout manager for 2D patches.

Tagged ,

invalid BS_jmpBoot in boot block: 000000

MediaFour‘s MacDrive (7.2.6) keeps screwing up the GPT on my external GPT-formatted drive. It has one HFS+ partition and one 200 MB EFI system partition, and when I boot my iMac into Windows (Vista SP1 x86), mount the drive, and then boot back into Mac OS X (10.5), I usually find that OS X won’t recognize the HFS+ volume any more. The characteristic error is an ignore/eject/format unrecognized dialog after OS X login, and “invalid BS_jmpBoot in boot block: 000000” from Disk Utility when trying to repair it.

It seems that the GUID for the HFS+ volume is changed from {48465300-0000-11AA-AA11-00306543ECAC} (HFS+) to {EBD0A0A2-B9E5-4433-87C0-68B6B72699C7} (“Microsoft Basic Data”). Windows doesn’t have any trouble mounting it as long as MacDrive is installed, it’s OS X that has the problem. I was able to fix the GPT by changing the GUID back to what it should be, at which point the drive was recognized by OS X, and found that the actual HFS+ file system was unharmed. I wrote most of a GPT editor in the process, and if Mediafour doesn’t fix MacDrive soon, I may end up turning it into a one-click GPT fixer.



Here’s the tool I used to fix my own disk: gpt_surgeon.py

It’ll need Python 2.5 or higher, and has only been tried in OS X on my own machine. Run it with no arguments or look at the code for a usage statement. The repair process goes like this:

First, list the partitions on your disk to figure out which one needs to be relabeled with the right filesystem type. If you don’t know what your disk device path is, go look in Disk Utility or System Profiler or run diskutil(8) with its list option. The path for my external disk is /dev/disk1.

$ ./gpt_surgeon.py list /dev/disk1

Read MBR and GPT from /dev/disk1.
partition 0:
     type: EFI System
     name: u'EFI System Partition'
    flags: 0x00000000
partition 1:
     type: Microsoft Basic Data
     name: u'Untitled'
    flags: 0x00000000

Note the partition that says “Microsoft Basic Data”. Assuming that you had a single non-system partition on the disk, and that the partition was HFS+ before MacDrive got to it, this is probably that partition. In my case, it’s partition 1. Remember the number; you’ll need it in the next step.

If you have multiple data partitions on the disk, especially if one of them is actually a Windows data partition, you should use something like disktype to check what filesystems are present in each partition, so that you don’t accidentally relabel the wrong one.

If you have any other partitions from this disk mounted, unmount them now.

You’ll need to run with sudo to get the privileges necessary to actually repair the GPT on disk.

$ sudo ./gpt_surgeon.py repair /dev/disk1 1

Read MBR and GPT from /dev/disk1.
Changing type of partition #1 on /dev/disk1 to HFS+...
    Opened /dev/disk1 for writing.
    Wrote MBR.
    Wrote GPT header.
    Wrote GPT entries.
    Closed /dev/disk1.

And this is what the disk’s GPT should look like afterwards. Your HFS+ partition should be mountable now.

$ ./gpt_surgeon.py list /dev/disk1

Read MBR and GPT from /dev/disk1.
partition 0:
     type: EFI System
     name: u'EFI System Partition'
    flags: 0x00000000
partition 1:
     type: Apple HFS+
     name: u'Untitled'
    flags: 0x00000000


second update: commenter James let me know that Christophe Grenier‘s TestDisk utility can also be used to fix this problem.


third update: here’s a video walkthrough of the process. For people who have Flash turned off for some reason, please see the QuickTime version of the walkthrough.


last update: gpt-surgeon is opening up! Thanks to all who submitted usage reports, suggestions, failure data, and proposed patches. I hope you’ll find future versions at least as useful. Comments have been turned off for this post; all questions, comments, suggestions, and support requests about gpt-surgeon must be submitted through the gpt-surgeon LaunchPad site and should be accompanied with disktype reports.

Tagged , ,