Today is a glorious day. First of all, the result of the last couple years of effort at Cycling ’74 was officially released. The early reception on Max 7 has been more than fabulous! Yay!
Not nearly as interesting, I have finally returned to InboxZero for the first time since my son was born over 4 years ago. That’s a bit embarrassing and it’s been draining, but everything left hanging out in my inbox was actually interesting (at least I didn’t keep the trash).
What made this process interesting was that I could see trends. The last three email threads were from different years, different contexts, and people. All were related to C++ code generation and on-the-fly compilation. This has been an area of extreme interest to me. The oldest email, from 2010, and perhaps the genesis of my interest in code generation was from the Plugtastic alpha testing.
Unless you know me personally and have a good memory, you probably don’t know about Plugtastic. Plugtastic took a Max patcher and exported it to an AudioUnit plug-in. The trick is that all of the objects needed to be Jamoma Graph and AudioGraph objects. So essentially it was a Jamoma AudioGraph using Max as the IDE.
When you exported the plug-in the graph was analyzed, C++ source code was generated from it, and then the code was compiled into the binary component.
Eventually, however, I was drawn in too many directions to keep the project going. Most of the initial work I did while on paternity leave after my son was born. But the demands of a second child, my work for Cycling ’74, continuing work on the core Jamoma infrastructure, and the need to engage my whole self (meaning physical activity and not just mental activity) ended up spreading me too thin. That said, it’s still a project that I find interesting and compelling.
The source code is now incorporated into Jamoma. If working on this project sounds interesting, that’s the place to look and to contribute your talents!
This past spring we rolled-out a new way compile our source code in the Jamoma project. After a number of months of real-world use in managing our code, the transition has been big success. I here propose that this system be moved beyond Jamoma for use by other software developers who care about portability, ease of use, ease of maintenance, and requisite flexibility.
The Maintenance Nightmare
We had been using Xcode projects on the Mac, Visual Studio on Windows, Makefiles on Linux, etc. The problem with maintaining all these independent projects is that a change in one requires changes in the others, which typically are changes that need to be made by someone running a different operating system. In the mean-time, the build is broken on that other platform. Or maybe instead the developer was extra conscientious and checked-in the change on a branch, using this work-flow:
create new branch
make change (add a file to an Xcode project)
check-in the change
check out the branch on a windows computer
make the change in visual studio
check-in the change
check out the branch on a linux computer
fix and test the Makefile
check-in the change
merge the branch to master
delete the temporary branch
delete the temporary branch on the Mac
delete the temporary branch on the Windows machine
delete the temporary branch on the Linux box
Realize that, oops, forgot to update the iOS target
In an open source project like Jamoma most of the work is done on an unpaid/volunteer basis. That means a lot of these steps don’t happen in a timely manner and the result is a project that is in a state of perpetual brokenness. (One side-effect of that state is that the project is hostile to potential new developers joining).
With our the new YAMLProj format, you simply add the file in the project on one platform, and it is added for all of the other platforms.
This is example is the YAMLProj for Jamoma’s TrajectoryLib extension.
To drive the point home, here is sequence of steps for adding a file to a project, as compared to the 16+ step behemoth presented at the outset:
make change (add a source file to the project)
check-in the change
No need to create a branch to prevent breaking the build on other platforms (though you could if you thought there were going to be problems).
Ease of Use
In addition to easing the maintenance burden, the system must also be easy for developers of all experience levels to use. Face it: not all developers are command-line junkies; a lot of people develop software by dragging files from the Mac Finder into Xcode and clicking the icon. That’s a good workflow if you want a non-intimidating experience that gets a quick result and makes you feel good.
Without writing an app for umpteen platforms, some of which wouldn’t even make sense, we’re not going to have drag-n-drop gui windows. Drag-n-drop gui windows can also be incredibly difficult to use and present the user with a complete cognitive cacophony that does not feel easy to use (e.g. Eclipse). So dispensing with that we have text files.
Text files can be intimidating. They have been used, misused, and abused in various ways. Some people even have baggage associated with them. The text file needs some sort of structure to it. Some text file formats for creating structure are hideous (ever looked at pbxproj file?), some are unforgiving (JSON), others are just tedious (XML). We chose YAML. It doesn’t require massive amounts of decoration, it is clear, it is easy and straightforward.
But what about CMake?
CMake sucks. There. I said it. I’ll upset some people. Whatever.
It’s too complicated. We don’t need our projects to fly the International Space Station. We need to compile a few source files and have it in a text file format that is not intimidating. In order to use CMake you probably have to compile some other open-source software, which might fail with errors that only the devs understand. Then you need it to work for compiling a lib for the iOS or some new platform that we haven’t heard about yet and you’re screwed.
Sorry. No CMake. We went down that road with Jamoma and threw it out in disgust. Don’t bother trying say anything contrary. I won’t be listening.
To the contrary, YAMLProj is super-simple, exceptionally clear, uncomplicated, and unintimidating. All the while still addressing the cross-platform maintenance issues in a reasonable way.
Making it Go
The implementation is a simple Ruby script. The Ruby script reads the YAMLProj, parses the YAML to generate the intermediate format and the compiles from the intermediate format. On Windows the intermediate format it generates is a VisualStudio project. On the Mac and Linux it generates a good old Makefile.
For completeness’ sake, here is the Makefile generated on a Mac for the aforementioned TrajectoryLib in Jamoma.
This is using the Apple Developer Tools-installed GCC version 4.2. If the Ruby script had detected a newer version of GCC or the Intel (ICC) compiler it would have used that instead. The compiler can be defined in the YAMLProj if a specific compiler is required or considered more desirable.
What about Editing and Debugging?
On Windows a VisualStudio project is created, so there’s nothing to worry about there. Developers on the Mac are used to being spoiled with the Xcode debugger and it first glance it looks like we’re throwing out Xcode projects here to use Makefiles.
In fact, we don’t have to throw out Xcode projects — we just need to change them a bit. Xcode projects can be configured to use as a source code editor that builds using the Makefile generated by our YAMLProj. It can then debug the project, complete with breakpoints.
Here is a quick video showing how to do it.
The existing implementation is primarily in a single Ruby method called generate_makefile which you can read on Github. The problem with this implementation is that it is tied directly to Jamoma. To be of general use it needs to be factored-out and made to standalone. It needs a group of people peer-reviewing it. It needs real-world use outside of Jamoma.
Max 6 introduced some big changes when it was released last fall. Flying under-the-radar were a number of new objects added to the collection. A few of these objects came from Tap.Tools, including the join/unjoin objects and the XmlParse class for MXJ. In the MSP world, the scale~ object entered the fray.
On the surface, scale~ is relatively straightforward. It maps one range onto another. With the addition of the exponent to provide non-linear functions the object becomes much more powerful, and little more difficult to quantify as a mathematical formula. More difficult still is a backwards-compatibility mode called “classic” which implements an idiosyncratic exponential base function from back in the ancient days of the ISPW. You probably don’t want to use this mode, but I’ve included it here for thoroughness sake.
So let’s take a technical look at scale~ beginning with the equation it implements.
This rendering was typeset in TextMate using the MacTex distribution of Latex. Here is the Latex source: