Pentaho

I sat in on a meeting to review the Pentaho tools. It's an impressive set of tools.

We focussed on the "Kettle" and "Spoon" tools (Pentaho uses a kitchen motif for their names). "Spoon" offers an easy-to-use interface for the configuration (it's too easy to be programming) of an ETL operation. The tools make for easy extraction, transformation, and loading (which is what ETL is all about).

Way back when, we called these tools "swabbers" (from "swap bytes A and B"). They were much simpler but did the same task: read data from a source, transform it, and send it to a destination. The Pentaho suite handles neat things like parallel threads and multiple sources and destinations and scheduled jobs.

Not a bad hour to invest.

Pair programming trade-offs

I attended the local cloud computing group meeting tonight. The presentation was on pair programming, one of the elements of agile development. The concepts were familiar to me, yet I took home a few thoughts:

Pair programming requires you to work with another person: You have to put aside your ego and listen to the other person. This idea goes back to at least the 1970s; Gerald Weinberg wrote about egoless programming in his "The Psychology of Computer Programming".

Pair programming affects productivity: Managers take note, because this affects the project schedule. With pair programming, time for development increases. It takes longer to implement the same features, compared to solo programming. (But not twice as long, oddly. The initial increase is sixty percent, but drops to twenty percent as people become comfortable with working in pairs.)

While the development time increases, so does the quality. The increased quality means fewer defects and therefore less time and effort for testing. Thus, pair programming shifts the effort from testing to development, with an overall reduction in effort (and time). The project manager must weigh the benefits and costs of this trade-off.

Recruiters and user groups

I chatted informally with a recruiter today. They informed that they had few openings for C++ in the area. I suspected that the demand for C++ would drop, but I thought it would be a gradual decrease, not the sharp drop we are seeing.

This evening I attended the BaltoMSDN group meeting. It was an open discussion about future technologies, with discussions on HTML5, Sharepoint, Silverlight, database administration, Gartner Group reports, and Windows Phone 7. It was a lively discussion, with lots of creative ideas. (This was a treat for me, as I can rarely attend these meetings.)

Agile Development presentation at CMAP

The CMAP group had a presentation on agile development techniques. For those familiar with agile development techniques, there were no surprises.

The key ideas from the presentation:

- Software development uses the metaphors of building construction, which serve us poorly. Software development is similar to a journey of exploration, not building a well-defined thing.

- The "big design up front" approach fails not from lack of planning or poor planning skills, but from our inability to predict the future. Project plans do not account for every possible event -- and never will.

- Allow for change and respond to it

- Revise the plan as you gain information

- Build features of the system, not layers. You can build a feature in layers, but the objective is to build the feature.

- Use feedback loops to ensure that you progress towards your goals

- Agile techniques are not tied to specific development tools

I was happy to see the CMAP group (a Microsoft-centered technology group) look at agile development.

OSCON 2011

I attended OSCON, the O'Reilly-run conference on open source software.

There were four "take-aways" from the conference:

Java has renewed interest, due to Oracle

Oracle purchased Sun Microsystems, and acquired Java in the process. Oracle has invested in Java, and the results show it. Java is not open source, and relations between Oracle and the open source community are uneasy, but as one person put it: "People are afraid of Oracle and its reputation of money-grubbing, hard-business operations. But it is precisely those money-grubbing, hard-business operations that allows Oracle the resources to invest in Java." Oracle has invested in Java. Recent changes to the JVM allow for dynamic languages, something that will enable open source projects.

Open source has matured to a respectable choice for development

Open source has changed its status. In the past, open source was a subversive movement, a "rebel alliance", against the empires of commercial software. That has changed. Open source is now a respected alternative for software. Its success with projects such as Apache, Linux, and Android have demonstrated that it can deliver workable solutions.

Open source is the research arm of the software industry

Open source has become the research laboratory for software. There are projects for large and small efforts, in various areas of software development. They explore new ideas and techniques, and release software quickly. Some recent accomplishments:

Scripting languages: Perl, Python, and Ruby are all open source projects. While Perl has been released slowly (version 6.0 has taken over a decade) the other languages have seen a number of revisions. The languages are capable and usable.

Future languages: There are open source implementations of a number of new languages. Some of these fall in the "functional programming" category, and offer efficiencies for cloud computing. The company that wants to get an early grasp of these languages can evaluate them today, thanks to open source.

Databases: The "NoSQL" data store concept was built in open source. This idea of a database abandons the SQL language and the relational organization of databases in exchange for performance and flexibility. NoSQL databases don't enforce a structure on the data and therefore can store varied information. They are often used for high-performance, large user base applications.

Distributed version control: Expanding version control for distributed projects, these DVCSs (distributed version control systems) allow people to work independently in distant locations. The DVCS coordinates updates from multiple people without the need for constant contact with the central database.

Open source is allowing companies to thrive in the current economic environment

Numbers can tell the story better than words. Of the two dozen exhibitor companies, over half were hiring. The hiring companies included small, obscure companies (Azur, Neustar, Percona, and Linbit) and large, well-known companies (the New York Times, O'Reilly, and Facebook). Some folks were at the conference for the express purpose of hiring -- no product demos, no sales pitches... just recruiting.

Hiring was competitive enough to spark a good-natured "recruitment war" during one of the closing sessions. Two companies (Grant Street Group and Booking.com) made multiple requests, advertising their hiring packages. Grant Street Group offered generous salaries and medical benefits; Booking.com offered to re-locate people to Amsterdam. No other companies joined the debate; perhaps they were overwhelmed by the competition.