[Mageia-sysadm] [205] version 1.9.1
root at mageia.org
root at mageia.org
Tue Jan 4 17:17:04 CET 2011
Revision: 205
Author: boklm
Date: 2011-01-04 17:17:04 +0100 (Tue, 04 Jan 2011)
Log Message:
-----------
version 1.9.1
Added Paths:
-----------
build_system/repsys/tags/1.9.1/
build_system/repsys/tags/1.9.1/CHANGES
build_system/repsys/tags/1.9.1/RepSys/binrepo.py
build_system/repsys/tags/1.9.1/repsys
Removed Paths:
-------------
build_system/repsys/tags/1.9.1/CHANGES
build_system/repsys/tags/1.9.1/RepSys/binrepo.py
build_system/repsys/tags/1.9.1/repsys
Deleted: build_system/repsys/tags/1.9.1/CHANGES
===================================================================
--- build_system/repsys/trunk/CHANGES 2011-01-04 16:09:44 UTC (rev 202)
+++ build_system/repsys/tags/1.9.1/CHANGES 2011-01-04 16:17:04 UTC (rev 205)
@@ -1,163 +0,0 @@
-* Development
-- added support to "binary repository"
-- added new commands upload, up and del to help handling tarballs in the
- binaries repository
-
-* 1.9
-- really fixed -M
-- new command log: shows the svn log for a given package
-- added option -s to co, to allow checking out only SPECS
-- use a better message when checking out from the mirror
-- show the path where getspec wrote the spec file
-
-* 1.8
-- make the -M option work again
-- sync now adds the spec file (pointed by blino)
-- fixed putsrpm to not create old log files for packages without changelog
-- submit now can fetch the revision number if not specified (it also shows
- the author and the first line of the commit)
-- allow submitting many packages at once, even without support on server
- side (the same behavior as running repsys submit for each package)
-- append a parameter sid=UUID for every set of packages submitted
-- strip username from package URL on submit (#53939)
-- clearer error message when svn co fails
-- svn authentication errors are handled, pointing to the wiki page on
- configuration
-- svn commands will not be shown on error messages, unless using --debug
-- repsys uses the wrapper repsys-ssh for svn, to handle signals and also
- set BatchMode
-- added the configuration option svn-env
-- show the error from rpm when fetching the version from a spec fails
-
-* 1.7
-- dropped all authenticated access support: subversion authentication has
- been broken for a long time and the workarounds weren't decent. It will
- be back in 1.7.x.
-- added configuration option svn-command in the global section, allowing
- to replace the default svn command
-- force svn+ssh:// URLs to be in BatchMode, in order to not have any
- interactivity at all with ssh
-- fixed incompatibility with Python-2.4 on urlparse
-- fixed emptylog message, which was not being shown when needed
-- distributions can be specified by using <distro>/<package> in all
- commands
-- allow submitting many packages at once (#28352)
-- package revisions in submit are now specified with name at nnn
-- the user can define groups of packages to be submitted in the section
- "submit-groups"
-- make 'repsys submit' without package name or revision number work again
-- added option --distro to submit
-- make putsrpm work again
-- added subcommand import as an alias to putsrpm
-- template: hide the first release when it has only invisible lines
-- added initial man page
-- allow resorting changelog entries through the config option sort in the
- log section
-- added rpmlog options: -o to append the old changelog, -p to append the
- changelog found in the spec, and -s to resort all changelog entries
-- rpmlog, getsrpm, getspec and changed will use the mirror if enabled
-- don't hide authors with only the first revision SILENTed (#41117)
-- fixed bad url used when using -v in getsrpm
-- if REPSYS_CONF is set, /etc/repsys.conf and ~/.repsys/config will not be
- readed anymore
-- sort the final changelog by enabling the option sort in the log section
-- merge the changelog found in the spec by enabling the option merge-spec
- in the log section
-- changed the built-in template to the current default.chlog
-- added option -d to repsys sync, to download the missing source files
-- added option -F to repsys ci to set a log message file
-- added option --strict to getsrpm to check if the revision provided
- matches the package URL;
-- changed the default command to build SRPMs to rpmbuild
-- added configuration option rpmbuild to the section helper, to define the
- command used to build packages
-- added boolean configuration option strict-revision in the submit
- section, to allow forcing the use of --strict
-- added option --list in create-srpm to list the available targets
-- make submit -l work on svn+ssh:// targets
-- the fix for the unreleased commits problem in the previous release was
- wrong, really fixed it
-- moved all configuration options that will hardly be changed to
- repsys-example.conf; we now have a shorter repsys.conf
-- fixed the use of file:/// URLs when using just the package name
-- allow using submit with package URLs having usernames
-- don't give the wrong message "invalid command 'CMD'" when this is not
- the case
-- added more help messages in subcommands
-
-* 1.6.19
-- added complement for SILENT: CLOG, which hides everything that does not
- start with this token
-- fixed generation of unreleased commits, it was using the previous
- markrelease revision as reference
-- added option -o to 'co' to disable the use of mirror when checking out
-
-* 1.6.18
-- added the subcommand "switch" to help with mirrors support
-- initialize plugins in create-srpm too
-- changelog: perform less svn calls to obtain release number and oldlog
-- changelog: show epoch even in the entry not released
-- changelog: make default.chlog compatible with cheetah-2
-- make "sync" compatible with rpm-4.4.8 behavior
-- "co" don't use mirror when URL is provided
-- "ci" don't relocate back to mirrors after commit (should use switch)
-- ldapusers: added options ldap-uri and ldap-starttls
-- fixed use of __import__, incompatible with python2.4 in plugin support
-- fixed bug of wrong paths when using mirrors
-
-* 1.6.17
-- brought from mdvsys world the sync command
-- ldapusers: the configuration format has changed, now it uses python
- template strings
-- ldapusers: many fixes: better error messages, ldap-port working, results
- contain only the fields needed, unbinding after search, filters are
- escaped
-
-* 1.6.16
-- introduced the plugin ldapusers: repsys user data obtained from LDAP;
- this plugin is builtin
-- added support to plugins, and the hability to wrap configuration sections
-- added workaround in the template to ignore empty releases
-- added initial support to mirrors, as requested by mrl; it required the
- new subcommand "ci"
-- changelogs from misc/ will come from HEAD and should be escaped (%%)
-
-* 1.6.15
-- empty changelog entries are now shown, with a EMPTYLOG tag to allow
- rpmlint warn the developer about it
-- check (and warn) if a temporary package has already been removed before
- trying to remove it
-
-* 1.6.2b
-- make submit pass --define options to create-srpm script
-- print error message when create-srpm fails
-- make get_srpm return the srpms list
-- add upload-srpm support in create-srpm
-
-* 1.6.2a
-- moved revision-offset to [log] section and added a comment
-
-* 1.6.2
-- reimplemented the option -n for rpmlog, which now uses the svn option
- --limit.
-- added the option revision-offset, for the Zero Day Revision issue.
-- small fix in main repsys help message
-
-* 1.6.1
-- added option url-map, as an workaround for svn+ssh:// urls problems
-- added configuration sectiom "helper"
-- added getsrpm-mdk and create-srpm to setup.py
-- added option -r to submit
-
-* 1.6.0
-- improved markrelease command line parsing
-- changelogs entries are now groupped by author, and sorted by revision
- number
-- the changelog now is generated using the Cheetah Template Engine, to
- allow quick modifications without spending time reading code and
- introducing new bugs
-- consequently, was added an option "-T <file>" to rpmlog and getsrpm to
- allow choosing the path of the template to be used
-- added options noauth=0, and baseurl=None in order to disable the
- authentication in some url schemes (http:// and file://)
-- replaced some "cl" references to "mdv"
Copied: build_system/repsys/tags/1.9.1/CHANGES (from rev 204, build_system/repsys/trunk/CHANGES)
===================================================================
--- build_system/repsys/tags/1.9.1/CHANGES (rev 0)
+++ build_system/repsys/tags/1.9.1/CHANGES 2011-01-04 16:17:04 UTC (rev 205)
@@ -0,0 +1,166 @@
+* 1.9.1-binrepo
+- fix problem with python threads on 2010.1
+
+* 1.9-binrepo
+- added support to "binary repository"
+- added new commands upload, up and del to help handling tarballs in the
+ binaries repository
+
+* 1.9
+- really fixed -M
+- new command log: shows the svn log for a given package
+- added option -s to co, to allow checking out only SPECS
+- use a better message when checking out from the mirror
+- show the path where getspec wrote the spec file
+
+* 1.8
+- make the -M option work again
+- sync now adds the spec file (pointed by blino)
+- fixed putsrpm to not create old log files for packages without changelog
+- submit now can fetch the revision number if not specified (it also shows
+ the author and the first line of the commit)
+- allow submitting many packages at once, even without support on server
+ side (the same behavior as running repsys submit for each package)
+- append a parameter sid=UUID for every set of packages submitted
+- strip username from package URL on submit (#53939)
+- clearer error message when svn co fails
+- svn authentication errors are handled, pointing to the wiki page on
+ configuration
+- svn commands will not be shown on error messages, unless using --debug
+- repsys uses the wrapper repsys-ssh for svn, to handle signals and also
+ set BatchMode
+- added the configuration option svn-env
+- show the error from rpm when fetching the version from a spec fails
+
+* 1.7
+- dropped all authenticated access support: subversion authentication has
+ been broken for a long time and the workarounds weren't decent. It will
+ be back in 1.7.x.
+- added configuration option svn-command in the global section, allowing
+ to replace the default svn command
+- force svn+ssh:// URLs to be in BatchMode, in order to not have any
+ interactivity at all with ssh
+- fixed incompatibility with Python-2.4 on urlparse
+- fixed emptylog message, which was not being shown when needed
+- distributions can be specified by using <distro>/<package> in all
+ commands
+- allow submitting many packages at once (#28352)
+- package revisions in submit are now specified with name at nnn
+- the user can define groups of packages to be submitted in the section
+ "submit-groups"
+- make 'repsys submit' without package name or revision number work again
+- added option --distro to submit
+- make putsrpm work again
+- added subcommand import as an alias to putsrpm
+- template: hide the first release when it has only invisible lines
+- added initial man page
+- allow resorting changelog entries through the config option sort in the
+ log section
+- added rpmlog options: -o to append the old changelog, -p to append the
+ changelog found in the spec, and -s to resort all changelog entries
+- rpmlog, getsrpm, getspec and changed will use the mirror if enabled
+- don't hide authors with only the first revision SILENTed (#41117)
+- fixed bad url used when using -v in getsrpm
+- if REPSYS_CONF is set, /etc/repsys.conf and ~/.repsys/config will not be
+ readed anymore
+- sort the final changelog by enabling the option sort in the log section
+- merge the changelog found in the spec by enabling the option merge-spec
+ in the log section
+- changed the built-in template to the current default.chlog
+- added option -d to repsys sync, to download the missing source files
+- added option -F to repsys ci to set a log message file
+- added option --strict to getsrpm to check if the revision provided
+ matches the package URL;
+- changed the default command to build SRPMs to rpmbuild
+- added configuration option rpmbuild to the section helper, to define the
+ command used to build packages
+- added boolean configuration option strict-revision in the submit
+ section, to allow forcing the use of --strict
+- added option --list in create-srpm to list the available targets
+- make submit -l work on svn+ssh:// targets
+- the fix for the unreleased commits problem in the previous release was
+ wrong, really fixed it
+- moved all configuration options that will hardly be changed to
+ repsys-example.conf; we now have a shorter repsys.conf
+- fixed the use of file:/// URLs when using just the package name
+- allow using submit with package URLs having usernames
+- don't give the wrong message "invalid command 'CMD'" when this is not
+ the case
+- added more help messages in subcommands
+
+* 1.6.19
+- added complement for SILENT: CLOG, which hides everything that does not
+ start with this token
+- fixed generation of unreleased commits, it was using the previous
+ markrelease revision as reference
+- added option -o to 'co' to disable the use of mirror when checking out
+
+* 1.6.18
+- added the subcommand "switch" to help with mirrors support
+- initialize plugins in create-srpm too
+- changelog: perform less svn calls to obtain release number and oldlog
+- changelog: show epoch even in the entry not released
+- changelog: make default.chlog compatible with cheetah-2
+- make "sync" compatible with rpm-4.4.8 behavior
+- "co" don't use mirror when URL is provided
+- "ci" don't relocate back to mirrors after commit (should use switch)
+- ldapusers: added options ldap-uri and ldap-starttls
+- fixed use of __import__, incompatible with python2.4 in plugin support
+- fixed bug of wrong paths when using mirrors
+
+* 1.6.17
+- brought from mdvsys world the sync command
+- ldapusers: the configuration format has changed, now it uses python
+ template strings
+- ldapusers: many fixes: better error messages, ldap-port working, results
+ contain only the fields needed, unbinding after search, filters are
+ escaped
+
+* 1.6.16
+- introduced the plugin ldapusers: repsys user data obtained from LDAP;
+ this plugin is builtin
+- added support to plugins, and the hability to wrap configuration sections
+- added workaround in the template to ignore empty releases
+- added initial support to mirrors, as requested by mrl; it required the
+ new subcommand "ci"
+- changelogs from misc/ will come from HEAD and should be escaped (%%)
+
+* 1.6.15
+- empty changelog entries are now shown, with a EMPTYLOG tag to allow
+ rpmlint warn the developer about it
+- check (and warn) if a temporary package has already been removed before
+ trying to remove it
+
+* 1.6.2b
+- make submit pass --define options to create-srpm script
+- print error message when create-srpm fails
+- make get_srpm return the srpms list
+- add upload-srpm support in create-srpm
+
+* 1.6.2a
+- moved revision-offset to [log] section and added a comment
+
+* 1.6.2
+- reimplemented the option -n for rpmlog, which now uses the svn option
+ --limit.
+- added the option revision-offset, for the Zero Day Revision issue.
+- small fix in main repsys help message
+
+* 1.6.1
+- added option url-map, as an workaround for svn+ssh:// urls problems
+- added configuration sectiom "helper"
+- added getsrpm-mdk and create-srpm to setup.py
+- added option -r to submit
+
+* 1.6.0
+- improved markrelease command line parsing
+- changelogs entries are now groupped by author, and sorted by revision
+ number
+- the changelog now is generated using the Cheetah Template Engine, to
+ allow quick modifications without spending time reading code and
+ introducing new bugs
+- consequently, was added an option "-T <file>" to rpmlog and getsrpm to
+ allow choosing the path of the template to be used
+- added options noauth=0, and baseurl=None in order to disable the
+ authentication in some url schemes (http:// and file://)
+- replaced some "cl" references to "mdv"
Deleted: build_system/repsys/tags/1.9.1/RepSys/binrepo.py
===================================================================
--- build_system/repsys/trunk/RepSys/binrepo.py 2011-01-04 16:09:44 UTC (rev 202)
+++ build_system/repsys/tags/1.9.1/RepSys/binrepo.py 2011-01-04 16:17:04 UTC (rev 205)
@@ -1,393 +0,0 @@
-from RepSys import Error, config, mirror, layout
-from RepSys.util import execcmd, rellink
-from RepSys.svn import SVN
-
-import sys
-import os
-import string
-import stat
-import shutil
-import re
-import tempfile
-import hashlib
-import urlparse
-import threading
-from cStringIO import StringIO
-
-DEFAULT_TARBALLS_REPO = "/tarballs"
-BINARIES_DIR_NAME = "SOURCES"
-BINARIES_CHECKOUT_NAME = "SOURCES-bin"
-
-PROP_USES_BINREPO = "mdv:uses-binrepo"
-PROP_BINREPO_REV = "mdv:binrepo-rev"
-
-BINREPOS_SECTION = "binrepos"
-
-SOURCES_FILE = "sha1.lst"
-
-class ChecksumError(Error):
- pass
-
-def svn_baseurl(target):
- svn = SVN()
- info = svn.info2(target)
- if info is None:
- # unversioned resource
- newtarget = os.path.dirname(target)
- info = svn.info2(newtarget)
- assert info is not None, "svn_basedir should not be used with a "\
- "non-versioned directory"
- root = info["Repository Root"]
- url = info["URL"]
- kind = info["Node Kind"]
- path = url[len(root):]
- if kind == "directory":
- return url
- basepath = os.path.dirname(path)
- baseurl = mirror.normalize_path(url + "/" + basepath)
- return baseurl
-
-def svn_root(target):
- svn = SVN()
- info = svn.info2(target)
- if info is None:
- newtarget = os.path.dirname(target)
- info = svn.info2(newtarget)
- assert info is not None
- return info["Repository Root"]
-
-def enabled(url):
- #TODO use information from url to find out whether we have a binrepo
- # available for this url
- use = config.getbool("global", "use-binaries-repository", False)
- return use
-
-def default_repo():
- base = config.get("global", "binaries-repository", None)
- if base is None:
- default_parent = config.get("global", "default_parent", None)
- if default_parent is None:
- raise Error, "no binaries-repository nor default_parent "\
- "configured"
- comps = urlparse.urlparse(default_parent)
- base = comps[1] + ":" + DEFAULT_TARBALLS_REPO
- return base
-
-def translate_url(url):
- url = mirror.normalize_path(url)
- main = mirror.normalize_path(layout.repository_url())
- subpath = url[len(main)+1:]
- # [binrepos]
- # updates/2009.0 = svn+ssh://svn.mandriva.com/svn/binrepo/20090/
- ## svn+ssh://svn.mandriva.com/svn/packages/2009.0/trafshow/current
- ## would translate to
- ## svn+ssh://svn.mandriva.com/svn/binrepo/20090/updates/trafshow/current/
- binbase = None
- if BINREPOS_SECTION in config.sections():
- for option, value in config.walk(BINREPOS_SECTION):
- if subpath.startswith(option):
- binbase = value
- break
- binurl = mirror._joinurl(binbase or default_repo(), subpath)
- return binurl
-
-def translate_topdir(path):
- """Returns the URL in the binrepo from a given path inside a SVN
- checkout directory.
-
- @path: if specified, returns a URL in the binrepo whose path is the
- same as the path inside the main repository.
- """
- baseurl = svn_baseurl(path)
- binurl = translate_url(baseurl)
- target = mirror.normalize_path(binurl)
- return target
-
-def is_binary(path):
- raw = config.get("binrepo", "upload-match",
- "\.(7z|Z|bin|bz2|cpio|db|deb|egg|gem|gz|jar|jisp|lzma|"\
- "pdf|pgn\\.gz|pk3|rpm|rpm|run|sdz|smzip|tar|tbz|"\
- "tbz2|tgz|ttf|uqm|wad|war|xar|xpi|zip)$")
- maxsize = config.getint("binrepo", "upload-match-size", "1048576") # 1MiB
- expr = re.compile(raw)
- name = os.path.basename(path)
- if expr.search(name):
- return True
- st = os.stat(path)
- if st[stat.ST_SIZE] >= maxsize:
- return True
- return False
-
-def find_binaries(paths):
- new = []
- for path in paths:
- if os.path.isdir(path):
- for name in os.listdir(path):
- fpath = os.path.join(path, name)
- if is_binary(fpath):
- new.append(fpath)
- else:
- if is_binary(path):
- new.append(path)
- return new
-
-def make_symlinks(source, dest):
- todo = []
- tomove = []
- for name in os.listdir(source):
- path = os.path.join(source, name)
- if not os.path.isdir(path) and not name.startswith("."):
- destpath = os.path.join(dest, name)
- linkpath = rellink(path, destpath)
- if os.path.exists(destpath):
- if (os.path.islink(destpath) and
- os.readlink(destpath) == linkpath):
- continue
- movepath = destpath + ".repsys-moved"
- if os.path.exists(movepath):
- raise Error, "cannot create symlink, %s already "\
- "exists (%s too)" % (destpath, movepath)
- tomove.append((destpath, movepath))
- todo.append((destpath, linkpath))
- for destpath, movepath in tomove:
- os.rename(destpath, movepath)
- for destpath, linkpath in todo:
- os.symlink(linkpath, destpath)
-
-def download(targetdir, pkgdirurl=None, export=False, show=True,
- revision=None, symlinks=True, check=False):
- assert not export or (export and pkgdirurl)
- svn = SVN()
- sourcespath = os.path.join(targetdir, "SOURCES")
- binpath = os.path.join(targetdir, BINARIES_CHECKOUT_NAME)
- if pkgdirurl:
- topurl = translate_url(pkgdirurl)
- else:
- topurl = translate_topdir(targetdir)
- binrev = None
- if revision:
- if pkgdirurl:
- binrev = mapped_revision(pkgdirurl, revision)
- else:
- binrev = mapped_revision(targetdir, revision, wc=True)
- binurl = mirror._joinurl(topurl, BINARIES_DIR_NAME)
- if export:
- svn.export(binurl, binpath, rev=binrev, show=show)
- else:
- svn.checkout(binurl, binpath, rev=binrev, show=show)
- if symlinks:
- make_symlinks(binpath, sourcespath)
- if check:
- check_sources(targetdir)
-
-def import_binaries(topdir, pkgname):
- """Import all binaries from a given package checkout
-
- (with pending svn adds)
-
- @topdir: the path to the svn checkout
- """
- svn = SVN()
- topurl = translate_topdir(topdir)
- sourcesdir = os.path.join(topdir, "SOURCES")
- bintopdir = tempfile.mktemp("repsys")
- try:
- svn.checkout(topurl, bintopdir)
- checkout = True
- except Error:
- bintopdir = tempfile.mkdtemp("repsys")
- checkout = False
- try:
- bindir = os.path.join(bintopdir, BINARIES_DIR_NAME)
- if not os.path.exists(bindir):
- if checkout:
- svn.mkdir(bindir)
- else:
- os.mkdir(bindir)
- binaries = find_binaries([sourcesdir])
- update = update_sources_threaded(topdir, added=binaries)
- for path in binaries:
- name = os.path.basename(path)
- binpath = os.path.join(bindir, name)
- os.rename(path, binpath)
- try:
- svn.remove(path)
- except Error:
- # file not tracked
- svn.revert(path)
- if checkout:
- svn.add(binpath)
- log = "imported binaries for %s" % pkgname
- if checkout:
- rev = svn.commit(bindir, log=log)
- else:
- rev = svn.import_(bintopdir, topurl, log=log)
- svn.propset(PROP_USES_BINREPO, "yes", topdir)
- svn.propset(PROP_BINREPO_REV, str(rev), topdir)
- update.join()
- svn.add(sources_path(topdir))
- finally:
- shutil.rmtree(bintopdir)
-
-def create_package_dirs(bintopdir):
- svn = SVN()
- binurl = mirror._joinurl(bintopdir, BINARIES_DIR_NAME)
- silent = config.get("log", "ignore-string", "SILENT")
- message = "%s: created binrepo package structure" % silent
- svn.mkdir(binurl, log=message, parents=True)
-
-def parse_sources(path):
- entries = {}
- f = open(path)
- for rawline in f:
- line = rawline.strip()
- try:
- sum, name = line.split(None, 1)
- except ValueError:
- # failed to unpack, line format error
- raise Error, "invalid line in sources file: %s" % rawline
- entries[name] = sum
- return entries
-
-def check_hash(path, sum):
- newsum = file_hash(path)
- if newsum != sum:
- raise ChecksumError, "different checksums for %s: expected %s, "\
- "but %s was found" % (path, sum, newsum)
-
-def check_sources(topdir):
- spath = sources_path(topdir)
- if not os.path.exists(spath):
- raise Error, "'%s' was not found" % spath
- entries = parse_sources(spath)
- for name, sum in entries.iteritems():
- fpath = os.path.join(topdir, "SOURCES", name)
- check_hash(fpath, sum)
-
-def file_hash(path):
- sum = hashlib.sha1()
- f = open(path)
- while True:
- block = f.read(4096)
- if not block:
- break
- sum.update(block)
- f.close()
- return sum.hexdigest()
-
-def sources_path(topdir):
- path = os.path.join(topdir, "SOURCES", SOURCES_FILE)
- return path
-
-def update_sources(topdir, added=[], removed=[]):
- path = sources_path(topdir)
- entries = {}
- if os.path.isfile(path):
- entries = parse_sources(path)
- f = open(path, "w") # open before calculating hashes
- for name in removed:
- entries.pop(removed)
- for added_path in added:
- name = os.path.basename(added_path)
- entries[name] = file_hash(added_path)
- for name in sorted(entries):
- f.write("%s %s\n" % (entries[name], name))
- f.close()
-
-def update_sources_threaded(*args, **kwargs):
- t = threading.Thread(target=update_sources, args=args, kwargs=kwargs)
- t.start()
- return t
-
-def upload(path, message=None):
- from RepSys.rpmutil import getpkgtopdir
- svn = SVN()
- if not os.path.exists(path):
- raise Error, "not found: %s" % path
- # XXX check if the path is under SOURCES/
- paths = find_binaries([path])
- if not paths:
- raise Error, "'%s' does not seem to have any tarballs" % path
- topdir = getpkgtopdir()
- bintopdir = translate_topdir(topdir)
- binurl = mirror._joinurl(bintopdir, BINARIES_DIR_NAME)
- sourcesdir = os.path.join(topdir, "SOURCES")
- bindir = os.path.join(topdir, BINARIES_CHECKOUT_NAME)
- silent = config.get("log", "ignore-string", "SILENT")
- if not os.path.exists(bindir):
- try:
- download(topdir, show=False)
- except Error:
- # possibly the package does not exist
- # (TODO check whether it is really a 'path not found' error)
- pass
- if not os.path.exists(bindir):
- create_package_dirs(bintopdir)
- svn.propset(PROP_USES_BINREPO, "yes", topdir)
- svn.commit(topdir, log="%s: created binrepo structure" % silent)
- download(topdir, show=False)
- for path in paths:
- if svn.info2(path):
- sys.stderr.write("'%s' is already tracked by svn, ignoring\n" %
- path)
- continue
- name = os.path.basename(path)
- binpath = os.path.join(bindir, name)
- os.rename(path, binpath)
- svn.add(binpath)
- if not message:
- message = "%s: new binary files %s" % (silent, " ".join(paths))
- make_symlinks(bindir, sourcesdir)
- update = update_sources_threaded(topdir, added=paths)
- rev = svn.commit(binpath, log=message)
- svn.propset(PROP_BINREPO_REV, str(rev), topdir)
- sources = sources_path(topdir)
- svn.add(sources)
- update.join()
- svn.commit(topdir + " " + sources, log=message, nonrecursive=True)
-
-def mapped_revision(target, revision, wc=False):
- """Maps a txtrepo revision to a binrepo datespec
-
- This datespec can is intended to be used by svn .. -r DATE.
-
- @target: a working copy path or a URL
- @revision: if target is a URL, the revision number used when fetching
- svn info
- @wc: if True indicates that 'target' must be interpreted as a
- the path of a svn working copy, otherwise it is handled as a URL
- """
- svn = SVN()
- binrev = None
- if wc:
- spath = sources_path(target)
- if os.path.exists(spath):
- infolines = svn.info(spath, xml=True)
- if infolines:
- rawinfo = "".join(infolines) # arg!
- found = re.search("<date>(.*?)</date>", rawinfo).groups()
- date = found[0]
- else:
- raise Error, "bogus 'svn info' for '%s'" % spath
- else:
- raise Error, "'%s' was not found" % spath
- else:
- url = mirror._joinurl(target, sources_path(""))
- date = svn.propget("svn:date", url, rev=revision, revprop=True)
- if not date:
- raise Error, "no valid date available for '%s'" % url
- binrev = "{%s}" % date
- return binrev
-
-def markrelease(sourceurl, releasesurl, version, release, revision):
- svn = SVN()
- binrev = mapped_revision(sourceurl, revision)
- binsource = translate_url(sourceurl)
- binreleases = translate_url(releasesurl)
- versiondir = mirror._joinurl(binreleases, version)
- dest = mirror._joinurl(versiondir, release)
- svn.mkdir(binreleases, noerror=1, log="created directory for releases")
- svn.mkdir(versiondir, noerror=1, log="created directory for version %s" % version)
- svn.copy(binsource, dest, rev=binrev,
- log="%%markrelease ver=%s rel=%s rev=%s binrev=%s" % (version, release,
- revision, binrev))
Copied: build_system/repsys/tags/1.9.1/RepSys/binrepo.py (from rev 204, build_system/repsys/trunk/RepSys/binrepo.py)
===================================================================
--- build_system/repsys/tags/1.9.1/RepSys/binrepo.py (rev 0)
+++ build_system/repsys/tags/1.9.1/RepSys/binrepo.py 2011-01-04 16:17:04 UTC (rev 205)
@@ -0,0 +1,394 @@
+from RepSys import Error, config, mirror, layout
+from RepSys.util import execcmd, rellink
+from RepSys.svn import SVN
+
+import sys
+import os
+import string
+import stat
+import shutil
+import re
+import tempfile
+import hashlib
+import urlparse
+import threading
+from cStringIO import StringIO
+
+DEFAULT_TARBALLS_REPO = "/tarballs"
+BINARIES_DIR_NAME = "SOURCES"
+BINARIES_CHECKOUT_NAME = "SOURCES-bin"
+
+PROP_USES_BINREPO = "mdv:uses-binrepo"
+PROP_BINREPO_REV = "mdv:binrepo-rev"
+
+BINREPOS_SECTION = "binrepos"
+
+SOURCES_FILE = "sha1.lst"
+
+class ChecksumError(Error):
+ pass
+
+def svn_baseurl(target):
+ svn = SVN()
+ info = svn.info2(target)
+ if info is None:
+ # unversioned resource
+ newtarget = os.path.dirname(target)
+ info = svn.info2(newtarget)
+ assert info is not None, "svn_basedir should not be used with a "\
+ "non-versioned directory"
+ root = info["Repository Root"]
+ url = info["URL"]
+ kind = info["Node Kind"]
+ path = url[len(root):]
+ if kind == "directory":
+ return url
+ basepath = os.path.dirname(path)
+ baseurl = mirror.normalize_path(url + "/" + basepath)
+ return baseurl
+
+def svn_root(target):
+ svn = SVN()
+ info = svn.info2(target)
+ if info is None:
+ newtarget = os.path.dirname(target)
+ info = svn.info2(newtarget)
+ assert info is not None
+ return info["Repository Root"]
+
+def enabled(url):
+ #TODO use information from url to find out whether we have a binrepo
+ # available for this url
+ use = config.getbool("global", "use-binaries-repository", False)
+ return use
+
+def default_repo():
+ base = config.get("global", "binaries-repository", None)
+ if base is None:
+ default_parent = config.get("global", "default_parent", None)
+ if default_parent is None:
+ raise Error, "no binaries-repository nor default_parent "\
+ "configured"
+ comps = urlparse.urlparse(default_parent)
+ base = comps[1] + ":" + DEFAULT_TARBALLS_REPO
+ return base
+
+def translate_url(url):
+ url = mirror.normalize_path(url)
+ main = mirror.normalize_path(layout.repository_url())
+ subpath = url[len(main)+1:]
+ # [binrepos]
+ # updates/2009.0 = svn+ssh://svn.mandriva.com/svn/binrepo/20090/
+ ## svn+ssh://svn.mandriva.com/svn/packages/2009.0/trafshow/current
+ ## would translate to
+ ## svn+ssh://svn.mandriva.com/svn/binrepo/20090/updates/trafshow/current/
+ binbase = None
+ if BINREPOS_SECTION in config.sections():
+ for option, value in config.walk(BINREPOS_SECTION):
+ if subpath.startswith(option):
+ binbase = value
+ break
+ binurl = mirror._joinurl(binbase or default_repo(), subpath)
+ return binurl
+
+def translate_topdir(path):
+ """Returns the URL in the binrepo from a given path inside a SVN
+ checkout directory.
+
+ @path: if specified, returns a URL in the binrepo whose path is the
+ same as the path inside the main repository.
+ """
+ baseurl = svn_baseurl(path)
+ binurl = translate_url(baseurl)
+ target = mirror.normalize_path(binurl)
+ return target
+
+def is_binary(path):
+ raw = config.get("binrepo", "upload-match",
+ "\.(7z|Z|bin|bz2|cpio|db|deb|egg|gem|gz|jar|jisp|lzma|"\
+ "pdf|pgn\\.gz|pk3|rpm|rpm|run|sdz|smzip|tar|tbz|"\
+ "tbz2|tgz|ttf|uqm|wad|war|xar|xpi|zip)$")
+ maxsize = config.getint("binrepo", "upload-match-size", "1048576") # 1MiB
+ expr = re.compile(raw)
+ name = os.path.basename(path)
+ if expr.search(name):
+ return True
+ st = os.stat(path)
+ if st[stat.ST_SIZE] >= maxsize:
+ return True
+ return False
+
+def find_binaries(paths):
+ new = []
+ for path in paths:
+ if os.path.isdir(path):
+ for name in os.listdir(path):
+ fpath = os.path.join(path, name)
+ if is_binary(fpath):
+ new.append(fpath)
+ else:
+ if is_binary(path):
+ new.append(path)
+ return new
+
+def make_symlinks(source, dest):
+ todo = []
+ tomove = []
+ for name in os.listdir(source):
+ path = os.path.join(source, name)
+ if not os.path.isdir(path) and not name.startswith("."):
+ destpath = os.path.join(dest, name)
+ linkpath = rellink(path, destpath)
+ if os.path.exists(destpath):
+ if (os.path.islink(destpath) and
+ os.readlink(destpath) == linkpath):
+ continue
+ movepath = destpath + ".repsys-moved"
+ if os.path.exists(movepath):
+ raise Error, "cannot create symlink, %s already "\
+ "exists (%s too)" % (destpath, movepath)
+ tomove.append((destpath, movepath))
+ todo.append((destpath, linkpath))
+ for destpath, movepath in tomove:
+ os.rename(destpath, movepath)
+ for destpath, linkpath in todo:
+ os.symlink(linkpath, destpath)
+
+def download(targetdir, pkgdirurl=None, export=False, show=True,
+ revision=None, symlinks=True, check=False):
+ assert not export or (export and pkgdirurl)
+ svn = SVN()
+ sourcespath = os.path.join(targetdir, "SOURCES")
+ binpath = os.path.join(targetdir, BINARIES_CHECKOUT_NAME)
+ if pkgdirurl:
+ topurl = translate_url(pkgdirurl)
+ else:
+ topurl = translate_topdir(targetdir)
+ binrev = None
+ if revision:
+ if pkgdirurl:
+ binrev = mapped_revision(pkgdirurl, revision)
+ else:
+ binrev = mapped_revision(targetdir, revision, wc=True)
+ binurl = mirror._joinurl(topurl, BINARIES_DIR_NAME)
+ if export:
+ svn.export(binurl, binpath, rev=binrev, show=show)
+ else:
+ svn.checkout(binurl, binpath, rev=binrev, show=show)
+ if symlinks:
+ make_symlinks(binpath, sourcespath)
+ if check:
+ check_sources(targetdir)
+
+def import_binaries(topdir, pkgname):
+ """Import all binaries from a given package checkout
+
+ (with pending svn adds)
+
+ @topdir: the path to the svn checkout
+ """
+ svn = SVN()
+ topurl = translate_topdir(topdir)
+ sourcesdir = os.path.join(topdir, "SOURCES")
+ bintopdir = tempfile.mktemp("repsys")
+ try:
+ svn.checkout(topurl, bintopdir)
+ checkout = True
+ except Error:
+ bintopdir = tempfile.mkdtemp("repsys")
+ checkout = False
+ try:
+ bindir = os.path.join(bintopdir, BINARIES_DIR_NAME)
+ if not os.path.exists(bindir):
+ if checkout:
+ svn.mkdir(bindir)
+ else:
+ os.mkdir(bindir)
+ binaries = find_binaries([sourcesdir])
+ update = update_sources_threaded(topdir, added=binaries)
+ for path in binaries:
+ name = os.path.basename(path)
+ binpath = os.path.join(bindir, name)
+ os.rename(path, binpath)
+ try:
+ svn.remove(path)
+ except Error:
+ # file not tracked
+ svn.revert(path)
+ if checkout:
+ svn.add(binpath)
+ log = "imported binaries for %s" % pkgname
+ if checkout:
+ rev = svn.commit(bindir, log=log)
+ else:
+ rev = svn.import_(bintopdir, topurl, log=log)
+ svn.propset(PROP_USES_BINREPO, "yes", topdir)
+ svn.propset(PROP_BINREPO_REV, str(rev), topdir)
+ update.join()
+ svn.add(sources_path(topdir))
+ finally:
+ shutil.rmtree(bintopdir)
+
+def create_package_dirs(bintopdir):
+ svn = SVN()
+ binurl = mirror._joinurl(bintopdir, BINARIES_DIR_NAME)
+ silent = config.get("log", "ignore-string", "SILENT")
+ message = "%s: created binrepo package structure" % silent
+ svn.mkdir(binurl, log=message, parents=True)
+
+def parse_sources(path):
+ entries = {}
+ f = open(path)
+ for rawline in f:
+ line = rawline.strip()
+ try:
+ sum, name = line.split(None, 1)
+ except ValueError:
+ # failed to unpack, line format error
+ raise Error, "invalid line in sources file: %s" % rawline
+ entries[name] = sum
+ return entries
+
+def check_hash(path, sum):
+ newsum = file_hash(path)
+ if newsum != sum:
+ raise ChecksumError, "different checksums for %s: expected %s, "\
+ "but %s was found" % (path, sum, newsum)
+
+def check_sources(topdir):
+ spath = sources_path(topdir)
+ if not os.path.exists(spath):
+ raise Error, "'%s' was not found" % spath
+ entries = parse_sources(spath)
+ for name, sum in entries.iteritems():
+ fpath = os.path.join(topdir, "SOURCES", name)
+ check_hash(fpath, sum)
+
+def file_hash(path):
+ sum = hashlib.sha1()
+ f = open(path)
+ while True:
+ block = f.read(4096)
+ if not block:
+ break
+ sum.update(block)
+ f.close()
+ return sum.hexdigest()
+
+def sources_path(topdir):
+ path = os.path.join(topdir, "SOURCES", SOURCES_FILE)
+ return path
+
+def update_sources(topdir, added=[], removed=[]):
+ path = sources_path(topdir)
+ entries = {}
+ if os.path.isfile(path):
+ entries = parse_sources(path)
+ f = open(path, "w") # open before calculating hashes
+ for name in removed:
+ entries.pop(removed)
+ for added_path in added:
+ name = os.path.basename(added_path)
+ entries[name] = file_hash(added_path)
+ for name in sorted(entries):
+ f.write("%s %s\n" % (entries[name], name))
+ f.close()
+
+def update_sources_threaded(*args, **kwargs):
+ t = threading.Thread(target=update_sources, args=args, kwargs=kwargs)
+ t.start()
+ t.join()
+ return t
+
+def upload(path, message=None):
+ from RepSys.rpmutil import getpkgtopdir
+ svn = SVN()
+ if not os.path.exists(path):
+ raise Error, "not found: %s" % path
+ # XXX check if the path is under SOURCES/
+ paths = find_binaries([path])
+ if not paths:
+ raise Error, "'%s' does not seem to have any tarballs" % path
+ topdir = getpkgtopdir()
+ bintopdir = translate_topdir(topdir)
+ binurl = mirror._joinurl(bintopdir, BINARIES_DIR_NAME)
+ sourcesdir = os.path.join(topdir, "SOURCES")
+ bindir = os.path.join(topdir, BINARIES_CHECKOUT_NAME)
+ silent = config.get("log", "ignore-string", "SILENT")
+ if not os.path.exists(bindir):
+ try:
+ download(topdir, show=False)
+ except Error:
+ # possibly the package does not exist
+ # (TODO check whether it is really a 'path not found' error)
+ pass
+ if not os.path.exists(bindir):
+ create_package_dirs(bintopdir)
+ svn.propset(PROP_USES_BINREPO, "yes", topdir)
+ svn.commit(topdir, log="%s: created binrepo structure" % silent)
+ download(topdir, show=False)
+ for path in paths:
+ if svn.info2(path):
+ sys.stderr.write("'%s' is already tracked by svn, ignoring\n" %
+ path)
+ continue
+ name = os.path.basename(path)
+ binpath = os.path.join(bindir, name)
+ os.rename(path, binpath)
+ svn.add(binpath)
+ if not message:
+ message = "%s: new binary files %s" % (silent, " ".join(paths))
+ make_symlinks(bindir, sourcesdir)
+ update = update_sources_threaded(topdir, added=paths)
+ rev = svn.commit(binpath, log=message)
+ svn.propset(PROP_BINREPO_REV, str(rev), topdir)
+ sources = sources_path(topdir)
+ svn.add(sources)
+ update.join()
+ svn.commit(topdir + " " + sources, log=message, nonrecursive=True)
+
+def mapped_revision(target, revision, wc=False):
+ """Maps a txtrepo revision to a binrepo datespec
+
+ This datespec can is intended to be used by svn .. -r DATE.
+
+ @target: a working copy path or a URL
+ @revision: if target is a URL, the revision number used when fetching
+ svn info
+ @wc: if True indicates that 'target' must be interpreted as a
+ the path of a svn working copy, otherwise it is handled as a URL
+ """
+ svn = SVN()
+ binrev = None
+ if wc:
+ spath = sources_path(target)
+ if os.path.exists(spath):
+ infolines = svn.info(spath, xml=True)
+ if infolines:
+ rawinfo = "".join(infolines) # arg!
+ found = re.search("<date>(.*?)</date>", rawinfo).groups()
+ date = found[0]
+ else:
+ raise Error, "bogus 'svn info' for '%s'" % spath
+ else:
+ raise Error, "'%s' was not found" % spath
+ else:
+ url = mirror._joinurl(target, sources_path(""))
+ date = svn.propget("svn:date", url, rev=revision, revprop=True)
+ if not date:
+ raise Error, "no valid date available for '%s'" % url
+ binrev = "{%s}" % date
+ return binrev
+
+def markrelease(sourceurl, releasesurl, version, release, revision):
+ svn = SVN()
+ binrev = mapped_revision(sourceurl, revision)
+ binsource = translate_url(sourceurl)
+ binreleases = translate_url(releasesurl)
+ versiondir = mirror._joinurl(binreleases, version)
+ dest = mirror._joinurl(versiondir, release)
+ svn.mkdir(binreleases, noerror=1, log="created directory for releases")
+ svn.mkdir(versiondir, noerror=1, log="created directory for version %s" % version)
+ svn.copy(binsource, dest, rev=binrev,
+ log="%%markrelease ver=%s rel=%s rev=%s binrev=%s" % (version, release,
+ revision, binrev))
Deleted: build_system/repsys/tags/1.9.1/repsys
===================================================================
--- build_system/repsys/trunk/repsys 2011-01-04 16:09:44 UTC (rev 202)
+++ build_system/repsys/tags/1.9.1/repsys 2011-01-04 16:17:04 UTC (rev 205)
@@ -1,93 +0,0 @@
-#!/usr/bin/python
-from RepSys import Error, plugins, config
-from RepSys.command import *
-import getopt
-import sys
-
-VERSION="1.9-binrepo"
-
-HELP = """\
-Usage: repsys COMMAND [COMMAND ARGUMENTS]
-
-Tool to access and manage a package repository structure.
-
-http://wiki.mandriva.com/en/Development/Packaging/RepositorySystem/Quickstart
-
-Useful commands:
- co checkout a package
- ci commit changes
- sync add-remove all file changes from the .spec
- submit submit a package for build
- putsrpm import a source package to the repository
- getspec prints the spec
- rpmlog prints the RPM changelog
- getsrpm creates the source RPM
- create create the structure of a new package
- changed shows changes not submitted
- authoremail prints the e-mail of a given author
- switch relocate to mirror or upstream repository
-
-Run "repsys COMMAND --help" and "man 8 repsys" for more information.
-
-Written by Gustavo Niemeyer <gustavo at niemeyer.net>
-"""
-
-command_aliases = {"import": "putsrpm"}
-
-def plugin_help(opt, val, parser, mode):
- if parser is None:
- prog = sys.argv[0]
- print "Use %s --help-plugin <plugin name>" % prog
- print "Available plugins:"
- print
- for name in plugins.list():
- print name
- else:
- print plugins.help(parser)
- raise SystemExit
-
-def parse_options():
- parser = OptionParser(help=HELP, version="%prog "+VERSION)
- parser.disable_interspersed_args()
- parser.add_option("--debug", action="store_true")
- parser.add_option("--help-plugins", action="callback", callback=plugin_help)
- parser.add_option("--help-plugin", type="string", dest="__ignore",
- action="callback", callback=plugin_help)
- opts, args = parser.parse_args()
- del opts.__ignore
- if len(args) < 1:
- parser.print_help(sys.stderr)
- sys.exit(1)
- opts.command = args[0]
- opts.argv = args
- return opts
-
-def dispatch_command(command, argv, debug=0):
- sys.argv = argv
- try:
- command = command_aliases[command]
- except KeyError:
- pass
- if debug:
- config.set("global", "verbose", "yes")
- try:
- repsys_module = __import__("RepSys.commands."+command)
- commands_module = getattr(repsys_module, "commands")
- command_module = getattr(commands_module, command)
- except (ImportError, AttributeError):
- etype, exc, tb = sys.exc_info()
- if tb.tb_next is None and not debug:
- raise Error, "invalid command '%s'" % command
- raise
- command_module.main()
-
-if __name__ == "__main__":
- try:
- plugins.load()
- except Error, e:
- sys.stderr.write("plugin initialization error: %s\n" % e)
- sys.exit(1)
- config.set("global", "repsys-cmd", sys.argv[0])
- do_command(parse_options, dispatch_command)
-
-# vim:et:ts=4:sw=4
Copied: build_system/repsys/tags/1.9.1/repsys (from rev 204, build_system/repsys/trunk/repsys)
===================================================================
--- build_system/repsys/tags/1.9.1/repsys (rev 0)
+++ build_system/repsys/tags/1.9.1/repsys 2011-01-04 16:17:04 UTC (rev 205)
@@ -0,0 +1,93 @@
+#!/usr/bin/python
+from RepSys import Error, plugins, config
+from RepSys.command import *
+import getopt
+import sys
+
+VERSION="1.9.1-binrepo"
+
+HELP = """\
+Usage: repsys COMMAND [COMMAND ARGUMENTS]
+
+Tool to access and manage a package repository structure.
+
+http://wiki.mandriva.com/en/Development/Packaging/RepositorySystem/Quickstart
+
+Useful commands:
+ co checkout a package
+ ci commit changes
+ sync add-remove all file changes from the .spec
+ submit submit a package for build
+ putsrpm import a source package to the repository
+ getspec prints the spec
+ rpmlog prints the RPM changelog
+ getsrpm creates the source RPM
+ create create the structure of a new package
+ changed shows changes not submitted
+ authoremail prints the e-mail of a given author
+ switch relocate to mirror or upstream repository
+
+Run "repsys COMMAND --help" and "man 8 repsys" for more information.
+
+Written by Gustavo Niemeyer <gustavo at niemeyer.net>
+"""
+
+command_aliases = {"import": "putsrpm"}
+
+def plugin_help(opt, val, parser, mode):
+ if parser is None:
+ prog = sys.argv[0]
+ print "Use %s --help-plugin <plugin name>" % prog
+ print "Available plugins:"
+ print
+ for name in plugins.list():
+ print name
+ else:
+ print plugins.help(parser)
+ raise SystemExit
+
+def parse_options():
+ parser = OptionParser(help=HELP, version="%prog "+VERSION)
+ parser.disable_interspersed_args()
+ parser.add_option("--debug", action="store_true")
+ parser.add_option("--help-plugins", action="callback", callback=plugin_help)
+ parser.add_option("--help-plugin", type="string", dest="__ignore",
+ action="callback", callback=plugin_help)
+ opts, args = parser.parse_args()
+ del opts.__ignore
+ if len(args) < 1:
+ parser.print_help(sys.stderr)
+ sys.exit(1)
+ opts.command = args[0]
+ opts.argv = args
+ return opts
+
+def dispatch_command(command, argv, debug=0):
+ sys.argv = argv
+ try:
+ command = command_aliases[command]
+ except KeyError:
+ pass
+ if debug:
+ config.set("global", "verbose", "yes")
+ try:
+ repsys_module = __import__("RepSys.commands."+command)
+ commands_module = getattr(repsys_module, "commands")
+ command_module = getattr(commands_module, command)
+ except (ImportError, AttributeError):
+ etype, exc, tb = sys.exc_info()
+ if tb.tb_next is None and not debug:
+ raise Error, "invalid command '%s'" % command
+ raise
+ command_module.main()
+
+if __name__ == "__main__":
+ try:
+ plugins.load()
+ except Error, e:
+ sys.stderr.write("plugin initialization error: %s\n" % e)
+ sys.exit(1)
+ config.set("global", "repsys-cmd", sys.argv[0])
+ do_command(parse_options, dispatch_command)
+
+# vim:et:ts=4:sw=4
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/mageia-sysadm/attachments/20110104/71eaa748/attachment-0001.html>
More information about the Mageia-sysadm
mailing list