ongoing

Feed title: | ongoing
Type: | atom
Encoding: | UTF-8

Feed: | ongoing
Item: | Cottage Life I: Logs

I think that life in general and this space in particular would benefit from more of an outdoor flavor; words and pictures rooted in Nature. Our recent acquisition of a piece of Keats Island, should make this easier. Welcome to Cottage Life. Any piece of Pacific Northwest waterfront is going to include a lot of logs.

A few trees naturally fall into the ocean when they die, but most of the logs that drift up on our beach represent little errors and omissions in the logging industry. Time was, you could make a living scooping these up and selling them back to the foresters; there was even a TV series about it.

I believe that the rock holding the log up represents the extreme northernmost point of Keats Island.

Here’s a close-up of another; they become more visually interesting as they spend time alternating between salt water and rocky beaches. I was walking along this one and was struck by the extreme whiteness of the exposed wood. I wonder how long it took?

Photo-geek note: we were on the island for five hours Saturday and I never felt even the slightest temptation to take the wide-angle prime lens off the camera.

Date: | 2008-06-16 07:00:17 UTC
Filed under: | The World/Cottage Life, The World, Cottage Life, Arts/Photos/Cameras, Arts, Photos, Cameras, The World/Places/Canada/Howe Sound, Places, Canada, Howe Sound

Feed: | ongoing
Item: | Deletionist Morons

For a nice safe NPOV (“Neutral Point of View”) discussion of the issues, see Deletionism and inclusionism in Wikipedia. My experience, which isn’t NPOV at all, is that Deletionists are knuckle-dragging droolers, walking vacant spaces, and as a side-effect generally, well, what’s the word I’m looking for? “Wrong.”

What got me involved was word that the Deletionist undead were shambling in the direction of _why’s entry. Oh, and by the way, apparently it’s somehow uncool that I entered the debate because I heard about it somewhere.

This one is obvious; _why is one of the handful of people who constitute the public face of Ruby. Some of his published code snippets make shivers run up and down my spine. Several of his libraries—most notably, I suppose, Hpricot—are very widely used. He’s written a popular online book. I’ve never met him but would very much like to.

Yes, he’s eccentric. Yes, he conceals his real name. Yes, he would almost certainly be happier if the entry were removed. So what? Wikipedia without that entry would be less accurate, less complete, and less useful.

In this particular case, the arguments from the deletionists are jargon-laden (hint: real experts use language that the people they’re talking to can understand), and either stupid or vacuous. Which is entirely consistent with my experience.

Are the deletionists simply, as it seems, reveling in their own inadequacies? This came up on Twitter last night and Jeff Atwood put it well: “The fatal flaw of deletionism is the mindset of deciding what someone else should find interesting.” Jeepers, what kind of impoverished soul does it take to delight in removing accurate and potentially-useful information from the permanent record based on the subjective and silly notion of “Notability”?

A little thought-experiment is in order: What harm would ensue were Wikipedia to contain an accurate if slightly boring entry on someone who was just an ordinary person and entirely fame-free? Well, Wikipedia’s “encyclopedia-ness” might be impaired... but I thought the purpose of Wikipedia was to serve the Net’s users, not worry about how closely it adheres to the traditional frameworks of the reference publishing industry?

I suggest the deletionist wankers go and cluster around an alternate online reference tome which has articles only about God, Immanuel Kant, and Britney Spears. Notability is not in question, so they should be happy.

Date: | 2008-06-15 21:34:36 UTC
Filed under: | Technology/Publishing/Reference, Technology, Publishing, Reference, The World/Life Online, The World, Life Online

Feed: | ongoing
Item: | Not Much

Two photos of not much in particular, but with explanations.

Explanation: Girls and Trucks.

This is another part of Main Street, which is partly explained in the first paragraph of Main Art.

I tried this last one in black and white, and it looked sort of stark and strong and lovely, but then I turned the colour back on and liked it better. I guess I’ll never be a Real Photographer.

Date: | 2008-06-13 07:32:18 UTC
Filed under: | Arts/Photos, Arts, Photos, The World/Places/Vancouver, The World, Places, Vancouver

Feed: | ongoing
Item: | Main Art

Our house is near, and my office on, Main Street, which despite its name is not, nor has it ever been, the main street of anything. These days “The Main” is starting to be hip and fashionable, but the process is hampered by the fact that the buildings are, on average, old, and mostly weren’t very good when they were built. So it was kind of nice when some street art popped up on one that I walk by every day.

It’s nice that somebody goes to the effort to do this when the results are so ephemeral. Whoever you are, thank you.

Date: | 2008-06-12 00:19:10 UTC
Filed under: | Arts/Photos, Arts, Photos, Arts/Visual, Visual

Feed: | ongoing
Item: | Cottagers!

Now we’re real Canadians. As of June 11th, Lauren and I own a cottage on Keats Island (Wikipedia, map, keatsisland.net). The consequences include a sudden interest in remote-area Internet options.

For context on the neighborhood, Canadian cottage culture, and pretty pix, check out Howe Sound day, which documents a trip we took mostly to look at the property. That was almost six months ago; time moves slower in cottage country. The two pictures in that piece identified as being from the north shore of Keats are taken from the cottage. Here’s another.

## Why?

When we looked at the place we liked the view, we liked the general setup, we liked the accessibility from Vancouver, the price seemed OK, and there was nothing obviously wrong.

While we were over there some of the party had a nap and I went for a walk in the woods with our son, now eight.

You have to understand that Keats is not a very big island, that the cottages are lined up along the oceanfront and that behind them the ground rises steeply into really rough bush. There were some rough paths through the bush behind the cottages so I said to the boy “let’s check ’em out.” We got maybe a hundred yards down the trail and he was saying “Dad, should we be here? Is it safe? Will we get lost? I think we should go back.” I realized that we were raising a 100% city boy who had probably never been in the woods, out of sight of human habitation, in his life. Suddenly the purchase seemed more attractive.

## Details

It’s pretty near the northernmost point of the island. It has an address—several of them in fact—but the streets named are purely hypothetical entries on some decades-old plan and were never built. If you zoom in close enough in Google maps you can actually see the place.

It’s got electricity and phone, and a very decent well shared with the other cottagers on that bit of shoreline. It has 150 feet of Pacific-ocean waterfront with a “foreshore lease” and a floating dock in the summer months, which is good because you can’t really drive to it. The house is probably 75 vertical feet above the water on a rocky cliff, so erosion worry is minimized and the view is good. There’s a weekend ferry that will drop you off at your own dock. There are two bedrooms and reasonably modern appliances. Plumbing is a problem.

## Internet!?!?

Well, there’d be dial-up access if that were any use. In the summer, I look forward to the occasional weekend completely off the grid, but that’s not very realistic if we wanted to, now and then, spend a week or more. Suffice it to say that the telco is not interested in extending DSL facilities to anywhere this remote.

Hmm... there are two companies competing to offer satellite Internet, but the data rates are pretty ordinary and it’s hundreds and hundreds of dollars to get installed and set up. Also, we’re at the north end of the island and there’s a small mountain in the middle, so I bet we can’t see the satellite.

We have direct line-of-sight to the “Hope Point Repeater” of the Gambier Radio Internet Project and its principal responds to email, so that might work.

We also have line-of-sight to most of Gibsons which means that if we could find someone there with broadband and an open mind, we could set up a long-distance WiFi repeater. Does anyone reading this live in Gibsons?

## Why Again

We tell ourselves that it’s a decent investment in times of financial turmoil, and that it’s for the kids, and those things are mostly true, but having a place where you can get up early and walk to say good morning to the ocean before you say anything to anyone else, that’s why really.

Date: | 2008-06-11 07:39:53 UTC
Filed under: | Arts/Photos, Arts, Photos, The World/Places/Canada/Howe Sound, The World, Places, Canada, Howe Sound

Feed: | ongoing
Item: | RX and 1.9 and Pain

This fragment is mostly a note to myself and placeholder and might prove useful to someone slashing through the XML undergrowth with bleeding-edge Ruby. Briefly: I revived my “RX” Ruby tokenizer (see here, here, and here) to contribute to Antonio Cangiano’s proposed Ruby benchmark suite, which I think is a Really Good Idea. I had a bit of pain getting the code to run on both Ruby 1.8 and 1.9, and then when I tried sanity-checking the output by comparing it to REXML on 1.9, REXML blew chunks. There are, apparently, issues about REXML and 1.9. Read on for details in the unlikely event that you care about any of this.

## Benchmarking

There’s this problem in that there are a lot of plausible-looking Ruby implementations now (MRI, YARV, JRuby, Rubinius, IronRuby, MagLev) and it would be nice to compare performance. I was talking to some of the implementers about this and someone (Charles Nutter I think) said “Problem is, there’s this huge gap between running `fib()` and running Rails.” So, for example, how do we find out how fast MagLev will run Rails, without going through all the pain of making MagLev run Rails?

Antonio Cangiano sensibly proposed Let’s create a Ruby Benchmark Suite; when Avi Bryant told me he’d tried my RX code on MagLev, it occurred to me that it might be an interesting benchmark.

> RX refresher: It’s a pure automaton-based XML tokenizer whose performance is totally dependent on the efficiency of dereferencing integer arrays, and it turns out that mainstream Ruby really sucks at this.

> To make it a little more competitive with REXML, the de-facto standard Ruby parser, I had kludged it all over the place with regex preprocessing to cut down on the array traffic.

So I asked Antonio whether, if I de-optimized RX to make it a pure array benchmark, would it be interesting for his suite, and he said yes, so I did.

## 1.8.6 vs. 1.9

Perhaps the single most visible difference between today’s Ruby and tomorrow’s is in the low-level string-handling API. Well, an XML parser lives entirely right there, so boy did I ever learn all about it. I had previously converted RX to run on 2006-vintage YARV, but I wanted one version of the code that would run in both 1.8.6 and 1.9. Sigh.

Here’s one of the detail issues, to give a feeling for the problems. Suppose you know that your input stream is in UTF-8 and you’ve read a buffer-full of data and you want to turn it into Unicode integer characters for the parser. The problem is that the buffer might end in the middle of a multi-byte UTF-8 character. Easy enough, a glance at the last byte will diagnose that. The problem is, how do you pull out the unsigned-integer value of the last byte of a buffer, without processing through the whole (potentially large) buffer, with code that runs in both Rubies?

I poked around on IRC and Eric Hodel managed to improve on my original suggestion. Read it and weep:

      def byte_at(s, i)
        s[i, 1].unpack('C')[0]
      end

## REXML Ouch

RX has a primitive unit-test suite; what I do to sanity-check it at a high level is feed a nontrivial XML doc to it and REXML and check that they find the same number of elements, PIs, paragraphs, `img` elements with a `src=` attribute whose value ends in `.jpg`, and occurrences of the word “the” in running text.

Well, when I finally got it running in Ruby 1.9, and started the sanity check, REXML blew up on my document, 2.8 Meg of the input to this blog.

With a bit of poking around, I ascertained that:

- REXML blew up differently depending whether I used its stream or DOM mode.

- In the stream mode, it apparently fell over while trying to handle an instance of `&#xbd;`, but I couldn’t replicate the failure with a small file containing that.

- In the DOM mode, it incorrectly reported an error on a CDATA section containing an XML declaration and DOCTYPE, but I couldn’t replicate the failure with a small file containing that.

- I asked Sam Ruby whether I should be surprised, and a glance at his continuous-integration setup for 1.9 reveals that I shouldn’t, there are lots of tests failing.

- There is quite a bit of disgruntlement about XML and Ruby right at this point in time; see The Status of Ruby’s libxml and My Frustrations with REXML: Ruby’s Standard Library for Reading/Writing XML; or, Ruby’s Problem Is Its Type System, and Don’t Try to Tell Me Otherwise and Does Ruby's support for XML suck? I need your input.

- The issue is under discussion on the ruby-core mailing list.

Well, there you go. By the way, in Ruby 1.9’s favor, it runs the (simplified de-optimized) RX about three times as fast as 1.8.6. Any other implementors want a whack at it?

Date: | 2008-06-11 06:29:20 UTC
Filed under: | Technology/XML, Technology, XML, Technology/Ruby, Ruby

Feed: | ongoing
Item: | Flamenco Sketches

Photographic sketches, I mean. Earlier this month, Eve and Eli were up from Seattle to visit and after dinner, we all dropped by the Kino Café to take in the flamenco. It was passionate and sexy and fun.

There was almost no stage lighting, and what there was in vivid reds and yellows; I pushed the Pentax/Sigma combo to the max, but it was tough.

There were four performers. The woman on the left was the flashier dancer, the white-haired fellow was a fine singer and still had a couple of moves left, and the guitar sounded as big as an orchestra.

But it was the other woman, something about her face, I just couldn’t take my eyes off it, beautiful yes, but so alive and engaged and in the moment.

The black-and white was an irritated reaction at my inability to colour-correct the seriously-unexposed picture, but then I started to like it. There are a couple in colour, one of the gentleman performer, and one of the dance solo. Not good pictures as such, but quite suggestive of what being there looked like.

The Kino ain’t fancy at all but it’s nice and friendly and the food and drink are just fine.

Oh, and Flamenco Sketches is the title of a really lovely tune, in a Spanish feel, on Miles Davis’ epochal Kind of Blue.

Date: | 2008-06-09 06:13:33 UTC
Filed under: | Arts/Music/Performance, Arts, Music, Performance, Arts/Photos, Photos

Feed: | ongoing
Item: | WF2: Early Results

[This is part of the Wide Finder 2 series.] The first serious results are in, and they’re eye-opening. The naive Ruby approach, remember, burned some 25 hours. There are now four other results posted, with elapsed times of 8, 9, 15, and 17 minutes. The write-ups are already full of surprises, and I’m expecting more.

Date: | 2008-06-09 05:16:29 UTC
Filed under: | Technology/Concurrency, Technology, Concurrency

Feed: | ongoing
Item: | Wide Finder 2

Last fall, I ran the Wide Finder Project. The results were interesting, but incomplete; it was a real shoestring operation. I think this line of work is interesting, so I’m restarting it. I’ve got a new computer and a new dataset, and anyone who’s interested can play.

## The Story Thus Far

I’ll use this entry as a table-of-contents for the series as it grows.

- The Benchmark.

- Benchmark Strawman.

- First Pathetic Results.

- People At Work.

- Start Your Engines!

- Early Results.

## The Problem

The problem is that lots of simple basic data-processing operations, in my case a simple Ruby script, run like crap on modern many-core processors. Since the whole world is heading in the slower/many-core direction, this is an unsatisfactory situation.

If you look at the results from last time, it’s obvious that there are solutions, but the ones we’ve seen so far impose an awful complexity cost on the programmer. The holy grail would be something that maximizes ratio of performance increase per core over programmer effort. My view: Anything that requires more than twice as much source code to take advantage of many-core is highly suspect.

## Last Time & This Time

There were a few problems last time. First, the disk wasn’t big enough and the sample data was too small (much smaller than the computer’s memory). Second, I could never get big Java programs to run properly on that system, something locked up and went weird. Finally, when I had trouble compiling other people’s code, I eventually ran out of patience and gave up. One consequence is that no C or C++ candidates ever ran successfully.

This time, we have sample data that’s larger than main memory and we have our own computer, and I’ll be willing to give anyone who’s seriously interested their own account to get on and fine-tune their own code.

## The Set-Up

This time, the computer is a T2000, with 8 cores and 32 threads. It’s actually slower (well, fewer cores) than the semi-pre-production T5120 I was working with last time, but it’s got over 250G of pretty fast disks, which I’ve built a ZFS pool and one filesystem on.

I’ve also staged all the ongoing logfiles from February 2003 to April 2008 there, 45G of data in 218 million lines.

It’s Internet-facing, with ssh access and port 80 open (not that I’m running a Web server yet).

## Want to Join In?

I’ve set up a WideFinder Project Wiki. The details of how to get started are there. For now, anyone with a wikis.sun.com account can write to it, which I’m hoping will be an adequate vandalism filter.

## The First Step

Before we start coding, we need to agree on the Benchmark; the 13-line Ruby program was an instructive target for Wide Finder 1, but several people pointed out that this could be done with an `awk`/`sort` one-liner, so something a little more ambitious might be appropriate. I’ve made a couple of initial suggestions on the wiki.

Date: | 2008-06-09 05:15:59 UTC
Filed under: | Technology/Concurrency, Technology, Concurrency

Feed: | ongoing
Item: | WF2: Start Your Engines!

[This is part of the Wide Finder 2 series.] I have now done the first “official” run of the naive Ruby implementation of the benchmark. There is some discussion of the code here. The benchmark is described, and the naive Ruby code provided, here. I’ve started a results page here. There are already ten eleven other people now with accounts on the Wide Finder machine, and I know there’ve been results that are hugely better than this first cut. Read on for a couple of notes on this first run.

The combination of using the SPARC-optimized Ruby and eliminating the huge sort shaved over four hours off the implementation time, but the results are still horrible: 25:24:41 elapsed time, with zero use of parallelism. The only thing that’s nice is the size: 78 lines of Ruby.

Can you do better on the time without exploding the code size?

An amusing note on the code. I needed a method to select the ten highest values (and corresponding keys) out of the hash table without sorting all of them. The code isn’t rocket science, here’s the skeleton:

    class Hash
      def top(howmany)
        ...
      end
    end
    # ... later on
    @clients = {}
    ...
    @clients[client] += 1
    ...
    top_clients = @clients.top(10)

I was showing it to a non-Rubyist and he said “WTF is that `class Hash` stuff?!” I explained and added, “Hmm, I guess I could have just said `def top(hash, howmany)`”. My questioner rolled his eyes. He had a point I think; Ruby makes you do some odd things.

Date: | 2008-06-06 07:43:53 UTC
Filed under: | Technology/Concurrency, Technology, Concurrency

Feed: | ongoing
Item: | Everything On-Line?

Recently I read The Rebellion Within by Lawrence Wright, a long, erudite, immensely informative New Yorker piece about the internal dynamics of Al-Qaeda and the Muslim Brotherhood and Islamic Jihad: Who are these people, anyhow, where did they come from, and where are they going? I entirely recommend it. There’s a problem, though: I read it on-line.

A high proportion of my reading time these days, both business and pleasure, is spent reading the Internet. But we still subscribe to The Economist and, whenever I’m about to get on a plane, I try to pick up a copy of the New Yorker; its articles are mostly very good, and usually there are one or two which are long and meaty; just the thing for an airplane ride.

The problem was, on a recent trip, that I glanced at the cover of the latest, bought it there at the airport, and discovered only after I was airborne that the big meaty piece was that radical-Islamist story.

Lacking alternatives, I read most of it again anyhow once my laptop battery was juiced out. The experience, on paper, was immensely better. This is not to diss the New Yorker’s online presentation, as good as any there is; the column width is about perfect, and the cartoons are inlined artfully in a way that doesn’t impede the flow of reading.

But for now, paper’s just way better for this sort of long-form piece. Maybe that’ll change. I’m not sure what I’m actually going to do about this; I’ll hardly pass up a juicy link that I have time to follow just because it’s also on sale in print.

It’s just that there’s part of me that wishes that the online and print worlds were perhaps a bit more cleanly separated.

Date: | 2008-06-06 05:05:15 UTC
Filed under: | Business/Publishing, Business, Publishing, The World/Life Online, The World, Life Online

Feed: | ongoing
Item: | San Fran Shadows

Last week I took a brief trip to San Francisco, and managed to escape for a walk. Lots of buildings had interesting shadows on them.

The old parts of San Francisco seem to age with grace, by and large.

Date: | 2008-06-05 05:23:35 UTC
Filed under: | The World/Places/San Francisco, The World, Places, San Francisco, Arts/Photos, Arts, Photos

Feed: | ongoing
Item: | Not an OS

Last Thursday I tweeted: “I strive to maintain an open mind when nontechnical people talk about the ‘Internet OS’ or ‘Web OS’. Sometimes it's tough.” I got some grumbles by email and I think the subject is worth more discussion. Let me be more specific: Neither the Internet nor the Web is much like an OS. And even if it were, that’d be the wrong way to think about what’s going on right now.

I’ve had this argument before. In March of 2007, Mike Arrington and I shared a stage at a startup event on the Sun campus, and talked about “Web 2.0”. Mike talked about a Web Operating System, I talked about people and information and business. Dan Farber reported ably. I don’t think either of us convinced each other.

## I: It’s About People Not Technology

The “Web OS” meme is harmful because it’s about technology. But the Internet’s killer app is people, has always been, will always be. Every single step forward has involved finding new routes and patterns and tools for people to use interacting with other people. No exceptions.

At the level of experience, Web 2.0 is about voices being heard. It used to be that there was a sanctified high priesthood that was licensed to publish: Journalists, novelists, and academics. Everyone else was an outsider, relegated to Letters-to-the-Editor or Speakers’ corner.

Nothing can possibly be more important than the experience of tens of millions of people, formerly outsiders, given a voice and an audience and a chance to join the global conversation.

This has been technically possible since there was an Internet. The crucial change was when Dave Winer and a few others proved that personal micro-publishing was fulfilling and self-supporting, by doing it.

Granted, a majority of people don’t publish—but if the world is even remotely like the population here at Sun, the knowledge that you’re empowered to do so is an attitude-changer.

Also granted, a majority of people who do publish don’t find big audiences. But that doesn’t matter; as Mena Trott was among the first to point out, “Publishing to fifteen people is perfectly OK if they’re the right fifteen people”.

## II: It’s About Information Not Technology

Or to use another label, the Long Tail. We’re in a world where we expect a significant amount of the information and inspiration and creativity and criticism and collaboration and stories and pictures and music and movies to come from the many at the edge, not just the few at the center. Yes, there will always be a loud-voiced few at the center, but they’re no longer alone.

The traffic patterns have changed forever and they’re not changing back. Language and its flows are arguably one of the defining characteristics of Homo sapiens. The impact of all this information redirection on who we are and what we do is central. To focus on the technology is to miss the important point.

## III: It’s About Business Not Technology

Lots of modern business is all about pumping information. The classic example would be finance; banks are giant information pumps with cash machines at the edges.

Organizations (business, governments, clubs, political parties, religions) who figure out how to surf the new information flow will succeed and prosper; those who push back will be swept away.

And it won’t have anything to do with whether anything’s like an OS or not.

## IV: Nobody Uses the OS Anyhow

Suppose the Net or Web were OS-like. So what? Nobody programs to one of those any more. I bet if I asked a survey of 100 high-level professional developers what was the last time they actually coded a Unix or Linux or Win32 system call, I’d get a blank stare.

Everybody I know is, as a matter of principle, working in a higher-level programming environment that goes to great lengths to abstract away the OS: Java, Ruby, Apache-project C with APR, you name it.

## V: It’s Platforms That Matter

When it comes to technology, what really matters is the platform you use to build with. Looking at your site from the outside, it’s not obvious to a non-expert (and perhaps not to an expert either) whether you built it with PHP or Rails or Django or JSF or ASP.NET.

But the choice of platform that you build with is incredibly important; in fact, most of the interesting new software innovations in the recent years have been in platform technology that let you build something good in less time with less pain that’s more maintainable.

The experience of building something with Rails is wildly different from that of a Java EE wrangler, which is wildly different still from that of someone in Microsoft-land. A common “Web OS”? Don’t see one.

In another sense, it’s reasonable to consider the Web as a platform; as Dave Winer memorably said “The platform that doesn’t have a vendor.” From the business and end-user and administrator point-of-view, this kind of works; but not from the seats where the people who build the apps sit.

In still another sense, it’s sane to refer to the browser as a platform; it’s certainly where a lot of code gets built. But it’s no more an OS than Gtk or WPF or Aqua is. It’s closest to the user but it’s a small piece of the big system.

The closest you’ll ever get to the essence of the Web-itself-as-platform is in working closely with HTTP and URIs, for example in implementing the Atompub protocol. But most developers working with Atompub will just use a handy library that fits into their Ruby or Java or C# ambiance, hardly even noticing.

## VI: And Anyhow, It’s Not Like an OS

The OS is the software that sits between you and the hardware. In practice, it offers a set of brutally stupid and complex services for managing storage and networking and the lowest level of user interaction. It’s difficult and unpleasant to use.

I’ve used several and even contributed in a small way to one or two. Neither the Internet nor the Web are anything like one in any important way. Thank goodness.

Date: | 2008-06-04 04:36:33 UTC
Filed under: | The World/Life Online, The World, Life Online, Business/Internet, Business, Internet, Technology/Internet, Technology, Technology/Web, Web

Feed: | ongoing
Item: | WF2: People At Work

[This is part of the Wide Finder 2 series.] I’m happy to report that I’ve given out a bunch of accounts on the Wide Finder 2 machine. I’ll aggregate links to others’ work in this entry.

## 2008/06/02

Ray Waldin has a bunch of posts: WideFinder in Scala, WiderFinder, WideFinder 2 Strawman implemented in Scala, Multicore WideFinding, and Multicore Sorter.

Alex Morega has Wide Finder, Wide Finder: Going Parallel, and Wide Finder: Many Cores.

## Did I Miss You?

If you’re working on this let me know and I’ll keep adding links.

Date: | 2008-06-03 15:16:51 UTC
Filed under: | Technology/Concurrency, Technology, Concurrency

Feed: | ongoing
Item: | Other Diddleys

Gosh, Bo Diddley’s gone. Which instantly reaffirmed one Internet lesson: if you want to boost the popularity of your YouTube videos, die. My two favorites (via Twitter today): Mona, with Tom Petty (listen to Bo’s guitar) and then this ancient B&W TV shoot with a young Bo and lots of screaming Sixties girls. But man, I love that rhythm, and lots of other people have played it well.

I’ve always smiled at George Thorogood’s take on Bo’s Who Do You Love?, but the YouTube pickings are kind of slim; this isn’t bad though it falls short of the version on 1978’s Move It On Over. But I personally think that Ronnie Hawkins owns that song, and have always loved the Last Waltz performance; for my money, one of Robbie Robertson’s best guitar outings ever.

And then there’s The Clash; around the time of London Calling they may or may not have been the Best Rock Band in the World, but to me at least, they were The Only Band That Mattered.

And on London Calling there were two bo-diddleys right after each other: Hateful and Rudie Can’t Fail. There’s a lot of Clash on YouTube, and watching some now I’m reminded of how unbelievably great they were live. But I can’t find either of those tunes. (Well, there’s a very decent Rudie from Joe Strummer and the Mescaleros). On the totally-unlikely off-chance that you like rock but don’t have London Calling, drop everything and buy it now.

Thanks for that rhythm, Bo, and those tunes built around it. There’ll be someone playing a bo-diddley in a bar somewhere as long as Homo sapiens draws breath.

Date: | 2008-06-03 06:08:08 UTC
Filed under: | Arts/Music, Arts, Music

Feed: | ongoing
Item: | Enterprisey Goodness

I tend to liking simple things, and to suspicion of big “enterprisey” software frameworks. I am dimly aware that up in the clouds there are platforms built on platforms built on platforms built on Java, towering edifices where acronyms like “JBI” and “ESB” and “SCA” live. Except for, I could never figure out what they actually, you know, did. Let’s be honest; the complexity and the high-level arm-waving about “Integration” scared me away and I never really tried. Well, I’ve stumbled into a closer look and am beginning to think there’s some there there.

I’ve been talking to Andi Egloff, who lives up there among the floating acronyms in these enterprisey clouds. I asked “Well, what does this software actually do?” After some talk he said “Let me show you what we’re working on.” Showing is so much better than telling.

They’re calling their work-in-progress “Fuji”. Here’s a screencast where they hook together an RSS feed, an XMPP stream, and a log-file, with an interposing a Ruby filter, using Fuji and a bit of DSL voodoo.

They’re also thinking of a visual interface; here’s the same logic, only with little boxes and arrows instead of DSLs. Personally I prefer the DSL.

It’s all based on OSGI, hmmm.

Now, in a world that already has Yahoo Pipes and simple Ruby POP-to-Twitter mashups, one wonders if you need a big honking Java howitzer to squish this particular ant.

So I asked Andi that, and he said “Yes, but then with JBI you can also talk to ESBs and other traditional enterprise integration points”. Which I’m not sure that YPipes or little Ruby scripts are built for. I asked him to show me that and he said they’re going to. I’m interested; stay tuned.

Date: | 2008-06-02 19:15:57 UTC
Filed under: | Technology/Java, Technology, Java, Technology/Web, Web

Feed: | ongoing
Item: | WF2: First Pathetic Results

[This is part of the Wide Finder 2 series.] I made a couple of little changes to the Strawman Benchmark and let ’er rip on the big 45Gig data set. The results were miserable, but instructive.

## Just How Miserable?

29:31:30 elapsed, 29:30:52 CPU, i.e. it maxed one of the T2000’s 32 CPUs. That’s one day plus five hours and change. Ouch!

## Small lessons

First of all, it was just using Ruby-by-default-out-of-the-box. As Haik pointed out, there’s a SPARC-optimized version available from Sun and I’ll install that.

Second, the code computes the various top-10 lists (hits, bytes, clients) by storing the numbers in a hash and running `Enumerable#sort_by`. It says right in the docs for that function that it’s inefficient. Still, it wasn’t too bad for sorting hashes keyed by ongoing resources, because there are only a few thousand of those. But when it sorted the hash keyed by the hostnames that had fetched resources, well... it wasn’t pretty. The `sort_by` call took hours.

It was made worse by a programming error on my part; several people pointed out that one couldn’t diff the program output because if several of the top entries had the same numeric value, they could show up in arbitrary order. I simply changed the code to sort on the value, then the key. Of course, I stupidly didn’t take out the old sort, so it ended up sorting twice.

Anyhow, I’ve changed the code to do a single `O(N)` scan through the big hashes, rather than sort them.

## Big Lessons

The naive Ruby code is making terrible use of the machine. Over the course of the run, it uses, on average, 3.2% of the available CPU resources, and 3.8% of the available memory.

If you look at the Bonnie results, this sucker can do simple block I/O at about 150M/second at a cost of about 75% of one CPU. So you ought to be able to plow through the 45G dataset in under five minutes, leaving 31 CPUs to do the actual work, even without fancy `mmap()` tricks. I bet somebody will write a WF2 implementation that runs in about five minutes as opposed to 29 hours.

## Next Steps

I’m going to re-run the cleaned-up Ruby code with the SPARC-optimized MRI and propose that as the baseline benchmark for WF2 implementations to beat. Then the fun should start.

Date: | 2008-05-31 21:24:46 UTC
Filed under: | Technology/Concurrency, Technology, Concurrency

Feed: | ongoing
Item: | Family Moment

The back porch door was open and our adorable little almost-two-year-old, all golden curls and pink dimples, was wandering in and out. One time as she was coming in, I noticed Marlowe the cat was sort of hovering around her; then the air was full of female shrieks: “She’s got a dead rat!” And so she did, holding it up all eager to please, while Marlowe looked confused and irritated. A fair-sized one too, with a good five inches of dangling tail. A few seconds later, she was just as confused and irritated as Marlowe, for the same reason, and the corpse was headed for disposal.

(Well, yes, we have two healthy cats that go outside, and this is a waterfront city thus will always have rats, and our cats are well-fed so they bring their prizes home to the back porch to show off.)

Date: | 2008-05-31 06:43:51 UTC
Filed under: | The World/Family, The World, Family, The World/Humor, Humor

Feed: | ongoing
Item: | WF2: The Benchmark

[This is part of the Wide Finder 2 series.] A bunch of people have requested the sample data. Meanwhile, over at the wiki, there’s a decent discussion going on of what benchmark we should run.

I particularly liked the comments from MenTaLguY, Preston Bannister, and Erik Engbrecht. (Unfortunately, given the test data we have, I don’t see us being able to follow Erik’s “non-ASCII” suggestion).

Of the suggestions on offer at the benchmark page, I prefer the last couple. Session statistics are a little trickier, and I suspect a bit more resistant to a brute-force Map/Reduce approach. And the “normal HTML report” idea is very much in the spirit of Wide Finder 1, only with a bit more computation involved, perhaps enough to keep it from being a pure parallel-I/O benchmark.

Your thoughts? I suppose I’m signed up to build a reference implementation for verifying output.

Date: | 2008-05-31 06:19:43 UTC
Filed under: | Technology/Concurrency, Technology, Concurrency

Feed: | ongoing
Item: | WF2: Benchmark Strawman

[This is part of the Wide Finder 2 series.] I’ve just updated the Wide Finder wiki with a straw-man benchmark (see “Strawman”), with sample Ruby code and output. Argh... it takes Ruby over 23 seconds to process just 100K lines. I hear talk of a Solaris/SPARC optimized version, gotta track that down. Comments please on the benchmark, then we can all start bashing our heads against the big file.

Date: | 2008-05-31 06:19:35 UTC
Filed under: | Technology/Concurrency, Technology, Concurrency