- You have a working knowledge of how to use a virtualenv.
- You have shell access to the turbogears site.
- You have permissions to update configuration and run builds on Jenkins
- You know how to run the nosetests on your local git clones of TurboGears.
- You have to have a working knowledge of git, and how to do merging and branching.
- You need permission to update the TurboGears2 and tg.devtools packages on PyPI
With those prerequisites in mind, this document will not cover all the command lines that you could run. Instead, we will specify steps as “activate your virtualenv” or “run nosetests” or the like.
Preparing a release of TurboGears is going to take some time and effort. It is unlikely to be completed in a mere day or two, so please plan on taking some time to work through it all.
The steps for the release, in summary, are as follows:
- Ticket System Triage
- Repository Branching
- Closing Remaining Tickets
- Upgrading All Local Packages As High As Possible
- Testing Jenkins With The Upgraded Packages And Code
- Finalizing Changes On ‘next’ Branch
- Preparing Changelog And Release Announcement
- Preparing Packages And The Documentation
- Uploading The Documentation
- Making The Source Distribution For The New Eggbasket
- Making The New Eggbasket The Current On Turbogears.org
- Pushing to PyPI
- Publishing Release Annoucement And Closing Milestones
- Final Cleanup
Below, we discuss each of these steps in detail.
Open the ticket system on SourceForge. Click “Edit Milestones”, and create the milestone that will be worked on after the milestone you are now working on releasing. For example, if you are working on release 2.1.1, you will create milestone 2.1.2. Also, make sure that there exists a major milestone after the current one. For example, if you are working on 2.1.1, make sure that 2.2.0 exists.
Next, you need to work your way through all of the open tickets on SourceForge and determine where they belong. This really does mean all open tickets. Tickets for the next major milestone could have been resolved in the current milestone. Tickets in the current milestone could be postponed to the next minor or major milestone. As a result, all open tickets must be visited and verified for which milestone they belong to.
Once this process has been completed, it would be wise to post an announcement to the TG ML and the TG-Dev ML informing people that any new tickets should only be placed against the next milestone (and what it is).
We have three main repositories: TG2, TG2Devtools, and TG2Docs. Each of them functions in a similar fashion for this process:
- Clone the repository
- Checkout the development branch
- Branch development and make a “next” branch
Once the “next” branch is made, this is where you will do all work until the release is done.
Working exclusively on the “next” branch in any of the relevant repositories, close out any remaining open tickets for this milestone. This particular step is likely to be the step that takes the greatest amount of time. To make it worse, we can’t give guidance in this document on how to do this. Each ticket is different.
The one thing we can say is this: Make sure you are running your tests. Jenkins will be watching any commits you push, and running tests. Don’t break him!
This part is fairly involved. As you will be uploading the result onto the TurboGears site, it is highly recommended that you have shell access to it.
Create a new virtual environment, and get “basketweaver” and “yolk” installed. You will need them both. They will be used later to find the eggs that can be updated, download the .tar.gz files, and prepare an eggbasket for public consumption.
Finally, you will need to make sure you have a copy of Python 2.4 and Python 2.6 installed and ready to work.
In order to mirror the current version of the eggbasket, the easiest way is to use a command similar to this:
user@host:~> rsync -avPL user@turbogears.org:/home/turbogearsorg/var/static/2.1/downloads/2.1/ ${HOME}/eggbasket/
user@host:~> chmod -R 0755 ${HOME}/eggbasket
Change the version from 2.1 to whatever it needs to be to correspond to the latest and greatest version of TurboGears.
In order to make the testing happen locally, you will need to make some updates to your system’s configuration. Edit your /etc/hosts file (or %SYSTEMDIR%etchosts.txt file on Windows), and add a reference that points www.turbogears.org to your local machine.
After doing this, update your web server to tell it that the proper path for the TurboGears files is your eggbasket. Assuming that your eggbasket is being stored at /home/user/eggbasket, and you are using Apache, a line similar to this (replacing the version as needed) in your httpd.conf will do the trick:
Alias /2.1/downloads/next /home/user/eggbasket
You are likely to need to restart your local webserver to make that work properly.
For TG2, TG2DevTools, and TG2Docs, you must make one change in each of them: setup.py has a “dependency_links” attribute. Change the word “current” to “next”, and commit the change. Don’t push the change to the world yet, though. You’re not ready for that just yet.
Make sure to change the template itself in tg2devtools/devtools/templates/turbogears/setup.py_tmpl !
For both Python 2.4 and Python 2.6, create a new virtualenv and run python setup.py tgdevelop and python setup.py tgdeps for each of the repositories.
Using the following commands, you will get your environment prepared with all possible packages.
user@host:~> cd ${HOME}/tg2
user@host:~> python setup.py nosetests
user@host:~> ls *.egg*
This will show you a complete list of the packages that were downloaded but not placed into your site-packages directory. Since the tool we use to scan for updated packages is only looking there, you need to remove all the local .egg files (except for the TurboGears2 .egg), and then “easy_install” the eggs you removed. Do this until all .egg files are replaced. Then, do the same for ${HOME}/tg2devtools .
Once done, the following commands should help you get new package versions for all packages that have upgrades, and get ready to install them. The following command will help you to upgrade all possible packages.
user@host:~> for pkg in `yolk -a -U | awk '{ print $1 }'` ; do echo Downloading ${pkg} ; yolk -F ${pkg}; done
user@host:~> makeindex *
Complete the process, looking in the virtual environment’s site-packages directory. Use yolk to download any/all packages you can, download the rest manually, and keep reiterating until yolk shows nothing more than pip, distribute, and (optionally) virtualenvwrapper.
Once you complete the process to get all packages as upgraded as possible, it’s time to start the testing. This is likely to be very easy. Make sure to test the installations with both Python 2.4 and Python 2.6. Use python setup.py nosetests to run them.
Now, at any point, if a package will not work, you need to either find a fix or revert to a previous version of that package. The choice must be made on a case by case basis.
Now that you have both Python 2.4 and Python 2.6 testing cleanly with the next branch locally, it’s time to take your eggbasket from your machine, and place it on the turbogears server. A command similar to this will help:
rsync -avP ${HOME}/eggbasket/ user@turbogears.org:eggbasket/
Once done, you will need to make it visible to the world under the downloads directory. Make sure that it matches the URL you placed in setup.py. Also pay close attention to the permissions when you do so, making sure that they are world-readable.
After doing this, visit Jenkins and update the build processes for the tg-next packages. Ideally, they will become very simple. Even still, verify all of the processes, and make sure that they work as expected.
Once done, you can finally do git push on all of the repositories. Run the actual builds for all of the tg-next packages, and make sure they come out clean. If so, the real work is done finally. The rest will only take you an hour or so. Otherwise, determine the problem, fix it, update the build process, git push (if applicable), and re-run the builds until they do come out clean.
As you go through the configuration on Jenkins, please remember this one very important thing: We are looking to make the installation process as easy as possible. Follow that guideline, so that we can make the process easier for our users.
After all the changes that you’ve made so far, the final changes are simply to get the new version numbers into the distributed files.
- In TG2:
- Update tg/release.py to have the new version number.
- Update the dependency_links in setup.py to reference the “current” URL instead of “next” URL.
- In TG2Devtools:
- Update setup.py:
- Update the version number
- Update the install requirements so that it requires TurboGears2 >= the new version number
- Update the dependency_links to reference the “current” URL instead of “next” URL.
- Update devtools/templates/turbogears/setup.py_tmpl:
- Update the dependency_links to reference the “current” URL instead of “next” URL.
- Update the install requirements so that it requires TurboGears2 >= the new version number
- In TG2Docs:
- Update book/conf.py and docs/conf.py:
- Update version and release to be the new version
- Update book/setup.py:
- Update the version number
- Update the dependency_links to reference the “current” URL instead of “next” URL.
Commit all of these changes, but do not push them public, not yet.
For each of the three repositories, you will need to review the commit logs since the last release. Gather up the summaries for each one, and prepare a new file. Use the standard GNU Changelog format. However, instead of recording individual file changes, record only the summaries. We don’t need the file changes since Git records those changes for us.
Review the SourceForge tickets for this milestone, and record any tickets that were closed for this repository but were not referenced in the summaries you’ve already recorded.
The changelog files you’ve made will be the commit message for the tags you are about to make.
In addition, prepare a release announcement. Anything I can say here sounds condescending. You should prepare it, though, so that as soon as you reach the “Publish” step, it’s all done in a few minutes.
First, merge the branch “next” onto the branch “master”. Then, tag the master branch with the new version number, and use the changelog you’ve generated as the commit message. The tag should be an annotated tag (i.e.: git tag -a").
Do this for each of the three repositories.
For the documentation, go into the appropriate directory, and type make html (either the docs or the book, whichever is needed to be uploaded).
When you run make html, it will create a directory “_build/html”. Upload the contents of that directory and replace the current directory with it. For instance, if you used rsync to upload to your user account on the server, and fixed the permissions so that the website user could read the files, you could then do rsync -avP --delete /path/to/new/docs /path/to/web/docs/directory and have everything properly uploaded/visible to the users.
At this point, everything is prepared, with one exception: The source distributions for TurboGears2 and tg.devtools must be placed in the eggbasket. Enter your local repository directory for both TG2.x Core and TG2.x DevTools and run python setup.py sdist. In both of them, you will produce a directory named dist with a .tar.gz file for the new version. Copy these files to your ${HOME}/eggbasket, then go to ${HOME}/eggbasket and run makeindex *.
Using the steps in Testing Jenkins With The Upgraded Packages And Code, upload the updated (and finalized) eggbasket to the turbogears.org web server.
Log in to the turbogears website. Go into the directory where you stored the “next” directory, and rename “next” to the version you are releasing. Remove the “current” link, and then do a symbolic link from the version being released to “current”, like so: ln -s 2.1.1 current
Publish your release announcement to the places of your choice. We recommend your blog(s) and twitter. In addition, update the turbogears “Current Status” page to reflect the new release.
For each of the three repositories, merge the “master” branch to the “development” branch.
You’re done. Sit back and enjoy having accomplished a release.