OSUNIX compared to OpenSolaris™ for core development

In this document we'll explain the fundamental differences between how development is handled between OSUNIX and OpenSolaris™.  To start we compare the tools they use and then later we compare the technical benefits, policy and practices behind them.

Core build tools - ospkg (fork of pkgcore)

ospkg, fork of pkgcore which is currently used by hundreds if not thousands of developers to build 10,000+ of the most popular open source software today.  With ospkg we're able to take advantage of all these existing build scripts and change only what's needed so that they build and install cleanly for OSUNIX.  Many popular packages include Xorg, KDE4.x, Perl, Python, Icedtea6 and everything else you could imagine from server to workstation.  We've changed pkgcore in a few important ways so that it tightly integrates with OSUNIX and smart package manager.  Below you'll find the highlighted differences as it compares to other build tools.

pkgcore vs pkgbuild vs gar vs conary

This chart isn't complete, but for those who know the corresponding tools it will be obvious how many features are missing and taken for granted once you haved used something powerful like pkgcore for years.

osunix-gate vs onnv-gate
onnv-gate is the vanilla upstream from Sun™ which requires you to sign the SCA (source contributor agreement) for your changes to be committed.  osunix-gate is our mirrored and merged hg repository that is externally controlled.  We have a putback and patch review process to ensure high quality engineering standards.  All patches must be OSI license compatible with the merging source and come with a promise that it will be maintained.  Before patches are directly applied we include them at the packaging level so they can undergo incubation period of testing and quality assurance.  This contrast to onnv-gate shows in our code review and community involvement process for testing.  Our testing and review process allows for greater exposure to diverse hardware, use cases and leveraging of the community.

How we build osunix-gate
From the bottom up we've built OSUNIX as an environment which makes a developers job easier.  To accomplish this we've removed all monolithic packaging and broken osunix-gate into logical divisions.  We assume some basic comparative knowledge about onnv-gate and how it's currently built.

The 30 second version of how onnv-gate is built.

Download onbld, closed binaries, hg clone onnv-gate, install undocumented packages dependencies via pkg, sign-up on sun.com and download Sun Studio 12.

>From onbld you'll have two tools and one script you need to modify depending on what you want to do.

You copy opensolaris.sh into the root source directory of onnv-gate, make changes and run ./bldevn -d opensolaris.sh && nightly

This fundamentally sounds easy, but currently there isn't any document which describes this entire process.  From beginning to end it's loosely coupled and doesn't follow any modern software development practice or packaging process.

Major faults in onnv-gate build process
1) opensolaris.sh isn't very well documented.
2) bldenv sets up 40+ env variables that affect hundreds of Makefile.master options and are only documented in code
3) Difficult to isolate build failures due to the size of code that's being built
4) Any patch or putback undergoes limited community testing exposure before being putback

Other problems with this process include building of large packages which should be maintained outside of onnv-gate.

We've solved most of these faults by creating logical divisions in packaging.

usr/src/lib/*
usr/src/cmd/*
usr/src/uts/*
usr/src/tools/*

For example in the hg repo we have the following directory

usr/src/cmd/ps/

This directory has a top level makefile which can be independently run to build a package

sys-cmds/ps

This direct logical connection between packaging and source allows for any developer to easily know where to find the code in order to make a patch or track down a bug.

Our build process is fully reproducible for any developer out of the box.

pmerge -1 sys-cmds/ps

This will pull the source, build tools, setup the build environment, build and optionally install the package or build a binary package.  All package dependencies have been resolved already and any new hg tag or revision can be used to build the entire core network and operating system.  This also keeps every file under centralized packaging and a single coherent packaging policy.

For developers wishing to work on code you can make a small change to the ps package script which will keep the source in a consistent location and give a single env script to build the correct env options for Makefile.master.  From there you can add your patches to the sys-cmds/ps/files directory and one line to the packaging script to include that patch in the package build process.  From there the patch can be uploaded, reviewed and shared by others.

We also specifically solve problems for allowing more fine grained build recipes and tuning of build options for different parts of the core os.  This can prove especially helpful for those needing to build and maintain appliances or during the toolchain upgrade process.  By focusing on critical areas first we can push and test with less effort and a greater level of quality assurance.

OpenSolaris and Sun are trademarks of Sun Microsystems   Inc in the US and respective areas.