Software Promotion

Once, a long time ago, I instituted a policy whereby any deployment of our internally developed systems was moved into production as source code. The installation scripts included building the system, as well as installing it. "Moving into production" is often called "deployment", but we now refer to it as software promotion because there are actually a whole series of environments that software gets moved into -- and only the last one is production. The set of environments varies a little, but usually consists of: systems test, integration test, performance test, user acceptance test, and (finally) production. All software traverses this gauntlet -- each move is a "promotion".

And ever since those early years, that software journey is always made by object code. Testing (and production) environments do not even include the build tools which would make it possible to work with the source. Linux vendors do the same. I.e. software is distributed as RPMs or DEBs which are binary. Yes, I know about Gentoo, and yes, the source for those RPMs or DEBs are available. But in the commercial universe, we prefer to deal with binaries -- even when the source is available. Never mind open source -- even our own source.

This doesn't just affect the act of software promotion. The development tools are not even available in the production (and test) environments. So the binaries are not even compiled with debugging information. You're not supposed to debug code in production. Or test. That's for development.

This is a deeply rooted bias. I worked briefly for an enterprise software company. During that time, I was trouble shooting a bug caused by a race condition between two threads. I eventually tracked it down -- and had a number of breakpoints set so that I could run the program, stop the two threads at just the right spots -- resume in the right order, and generate the bug -- breaking right where one could inspect the faulty values. I walked over to the cube for the developer responsible for that module and asked him to walk back to mine so I could show him the sequence.

"E-mail me the logs", he said. Try as I might, I couldn't get him to take a look at the debugger. That's just not the way it's done. Problems are found by examining the logs. Subtle problems are found by cranking up the log level -- generating more verbose logs -- and examining the logs more intently. Developers look at source code -- during development. Once you move into testing and production, everybody (developers and administrators and operators) looks at logs.

Why is that? Why is software promotion (or deployment) such a binary affair? Why don't we have compilers and build tools and debuggers installed in production?

Certainly, in a large IT environment, the whole point is to have the software in production -- and to operate and trouble-shoot it there. If it is standard practice in most system environments (production and test) to *not* have the source -- or any of the source-related tools -- what does that say about the significance or importance of access to the source code for operations or testing?

So, if I'm not developing an operating system (or database, or application server) -- but merely testing one, or running it, and my IT organization doesn't use source code in test or production for even internally developed applications, why would an open source system be any different than a closed source one? And if it should be different, how would I use the source in production? Or even testing?