diff --git a/branches/pisi-1.1/AUTHORS b/branches/pisi-1.1/AUTHORS new file mode 100644 index 00000000..e1af7c46 --- /dev/null +++ b/branches/pisi-1.1/AUTHORS @@ -0,0 +1,22 @@ +Current maintainer +------------------ +Faik Uygur + +Authors and contributors +------------------------ + +Ahmet Aygun +A. Murat Eren +Bahadır Kandemir +Barış Metin +Faik Uygur +Furkan Duman +Görkem Çetin +Gürer Özen +İsmail Dönmez +Onur Küçük +S. Çağlar Onur + +Inactive contributors +--------------------------------- +Eray Özkural < erayo@cs.bilkent.edu.tr > diff --git a/branches/pisi-1.1/CODING b/branches/pisi-1.1/CODING new file mode 100644 index 00000000..d47b9e41 --- /dev/null +++ b/branches/pisi-1.1/CODING @@ -0,0 +1,150 @@ +Like every serious project, there are guidelines. +"Coding Standards" for serious. + +Guidelines +---------- + +0. Before reading any further please observe + PEP 8: Style Guide for Python Code + http://www.python.org/peps/pep-0008.html + + In particular this means no lameCaps + +1. When using dirnames, don't expect the dir to end + with a trailing slash, and please use the dirnames + in pisiconfig. Use util.join_path instead of os.path.join +2. Python indentation is usually 4 spaces. +3. Follow python philosophy of 'batteries included' +4. Use exceptions, don't return error codes +5. Don't make the PISI code have runtime dependencies on + a particular distribution (as much as possible). +6. Don't assume narrow use cases. Allow for a mediocre + amount of generalization in your code, for pieces that + will be required later. +7. If you are changing something, check if that change + breaks anything and fix breakage. For instance a + name. Running the tests is not always enough! +8. A good design ensures separation of concerns. Every module + has a specific documented responsibility. Don't make the + horse clean your windows. +9. To ensure readability avoid nesting python constructs + more than 3 levels deep. Python is a good language (unlike C), + so you can define inner functions in a convenient way, use + such decomposition techniques to break down your code into + manageable chunks. The worst code you can write is one huge + procedure that goes on for 1000 (or more) lines. +10. Use a particular abstraction like a class or function only + if it makes sense. Don't just define things because they can + be defined. Define only things that will/may be used. +11. If you are doing an expensive task like searching through + 10000 text chunks, please use an efficient data structure + and algorithm. We are not MS engineers who know no data + structure beyond doubly linked lists and no algorithm beyond + quicksort. +12. Resist the temptation to develop kludges and workarounds in + response to pressure. Take your time to solve the problems by + the book. The payoff comes later. +13. Same thing goes for premature optimizations. Knuth and Dijkstra + are watching over your shoulder. :) + +Branches and SVN +---------------- + +There are two branches of pisi, one is called pisi-devel and +new features that are large enough to cause instability go +into that branch. The trunk version is supposed to be stable at +all times. This means that you *must* run unit tests and other +test scripts after committing any change that cannot be tested +in isolation. Run the unit tests periodically to catch unseen +bugs. A release from the stable branch *must not* break any tests +whatsoever, so extensive use of the test suite must precede any +release. + + +Unit testing +------------ + +Unit tests are located in unittests directory. Running the tests is +trivial. But you must synchronize your code and data with the test +code, which can be a tedious work if you lose discipline. + +Sample data files are located in the same directory with test modules. + +For running the entire test suite, use the following command: + +$ ./tests/run.py + +The following command will run tests in specfiletests and archivetests +in unittests dir: + +$ ./tests/run.py specfile archive + +Do not depend on the output of unittests. Instead of producing an +output message/data in your tests, check the data internally. By +definition, unittest should just report succeeding and failing cases. + +If you didn't, take a look at the links below for having an idea of +unit testing. +http://www.extremeprogramming.org/rules/unittests.html +http://www.extremeprogramming.org/rules/unittests2.html + + +Other tests +----------- + +There are a couple of nice test scripts for testing the basic +capabilities of the command line interface such as building and +upgrading. Unlike unit tests, you have to take a look at the output +to understand that the scripts are doing well :) + +Misc. Suggestions +----------------- + +1. Demeter's Law + +In OO programming, try to invoke Demeter's law. +One of the "rules" there is not directly accessing any +objects that are further than, 2/3 refs, away. So the +following code is OK. + destroy_system(a.system().name()) +but the following isn't as robust + destroy_system(object_store.root().a.system.name()) +As you can tell, this introduces too many implementation +dependencies. The rule of thumb is that, in these cases +this statement must have been elsewhere.... It may be a +good idea to not count the object scope in this case, +so in Python self.a means only one level of reference, +not two. + +One quibble with this: it may be preferable not to insist +on this where it would be inefficient. So if everything +is neatly packed into one object contained in another +object, why replicate everything in the upper level? If +the semantics prevents dependency changes, then chains +of 3 or even 4 could be acceptable. + +OTOH, in Python and C++, it's not always good to implement +accessor/modifier pairs for every property of an object. +It would be much simpler if you are not doing any special +processing on the property (e.g. if what the type system +does is sufficient). + +The main rule of thumb in Demeter's Law is avoiding +putting more than, say, 10 methods in a class. That works +really well in practice, forcing refactoring every now +and then. + +2. We all know, you're using LISP but didn't want to tell +us. Don't be scared, as a success story and for your encouragment +there are tens of people somewhere with LISP releated jobs. + +3. If you are studying Data structures and Algorithms, and if +your first assignment is to implement a basic FIFO queue, +don't implement it. Just show your teacher the syntax of LISP, +tell him how beautiful it is, and show how an autistic person +can count lots of parenthesis with a "one second" look, you'll +probably get A+. + +4. If you are interested in "Playstation 2 Linux Games Programming" +or "How to extend C programs with Guile", please don't exercise +your valuable skills in this project. diff --git a/branches/pisi-1.1/COPYING b/branches/pisi-1.1/COPYING new file mode 100644 index 00000000..059bf0ed --- /dev/null +++ b/branches/pisi-1.1/COPYING @@ -0,0 +1,346 @@ +NOTE! The GPL below is copyrighted by the Free Software Foundation, but +the instance of code that it refers to (the kde programs) are copyrighted +by the authors who actually wrote it. + +--------------------------------------------------------------------------- + + GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc. + 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Library General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. + + GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. + + 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) 19yy + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + + +Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this +when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) 19yy name of author + Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, the commands you use may +be called something other than `show w' and `show c'; they could even be +mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program + `Gnomovision' (which makes passes at compilers) written by James Hacker. + + , 1 April 1989 + Ty Coon, President of Vice + +This General Public License does not permit incorporating your program into +proprietary programs. If your program is a subroutine library, you may +consider it more useful to permit linking proprietary applications with the +library. If this is what you want to do, use the GNU Library General +Public License instead of this License. diff --git a/branches/pisi-1.1/INSTALL b/branches/pisi-1.1/INSTALL new file mode 100644 index 00000000..33a2f00c --- /dev/null +++ b/branches/pisi-1.1/INSTALL @@ -0,0 +1,16 @@ +You can install PiSi on your system with a single command. + +# ./setup.py install + +If you are on a Pardus system, you should delete /usr/lib/pardus/pisi +when you are installing the SVN version + +PiSi requires piksemel XML processing library, the newest version +bsd bindings called "bsddb3" and Pardus configuration manager COMAR +in order to run. + +If you are upgrading from an old PiSi release you may have to +run the following command as root: + +# pisi rebuild-db + diff --git a/branches/pisi-1.1/MANIFEST.in b/branches/pisi-1.1/MANIFEST.in new file mode 100644 index 00000000..e423d286 --- /dev/null +++ b/branches/pisi-1.1/MANIFEST.in @@ -0,0 +1,7 @@ +recursive-include po * +recursive-include tests *.py +recursive-include tools * +recursive-include doc * +include . *.dtd +include . README NEWS CODING COPYING + diff --git a/branches/pisi-1.1/README b/branches/pisi-1.1/README new file mode 100644 index 00000000..d1d026d1 --- /dev/null +++ b/branches/pisi-1.1/README @@ -0,0 +1,18 @@ +PISI - Packages Installed Succesfully as Intended + +PISI is a new package manager for the PARDUS +distribution. In Turkish PISI means "kitty", and +like a kitty, it is featureful and small. + +Some of its distinctive features: + + - Implemented in python + - Efficient and small + - Package sources are written in XML and python + - Uses LZMA for a better compression ratio + - Fast database access implemented with berkeley DB + - Integrates low-level and high-level package operations (dependency resolution) + - Framework approach to build applications and tools upon + - Comprehensive CLI and a user-friendly qt GUI (distributed separately) + - Extremely simple package construction + diff --git a/branches/pisi-1.1/README.upgrade b/branches/pisi-1.1/README.upgrade new file mode 100644 index 00000000..a0577caa --- /dev/null +++ b/branches/pisi-1.1/README.upgrade @@ -0,0 +1,5 @@ +For upgrading from pisi 1.0 and before + +You have to move the packages in /var/lib/pisi under /var/lib/pisi/package and remove /var/db/pisi/ directory. + +db version has changed, so a rebuild-db will be unfortunately necessary. diff --git a/branches/pisi-1.1/doc/algorithm.sty b/branches/pisi-1.1/doc/algorithm.sty new file mode 100644 index 00000000..843e3d5b --- /dev/null +++ b/branches/pisi-1.1/doc/algorithm.sty @@ -0,0 +1,79 @@ +% ALGORITHM STYLE -- Released 8 April 1996 +% for LaTeX-2e +% Copyright -- 1994 Peter Williams +% E-mail Peter.Williams@dsto.defence.gov.au +\NeedsTeXFormat{LaTeX2e} +\ProvidesPackage{algorithm} +\typeout{Document Style `algorithm' - floating environment} + +\RequirePackage{float} +\RequirePackage{ifthen} +\newcommand{\ALG@within}{nothing} +\newboolean{ALG@within} +\setboolean{ALG@within}{false} +\newcommand{\ALG@floatstyle}{ruled} +\newcommand{\ALG@name}{Algorithm} +\newcommand{\listalgorithmname}{List of \ALG@name s} + +% Declare Options +% first appearance +\DeclareOption{plain}{ + \renewcommand{\ALG@floatstyle}{plain} +} +\DeclareOption{ruled}{ + \renewcommand{\ALG@floatstyle}{ruled} +} +\DeclareOption{boxed}{ + \renewcommand{\ALG@floatstyle}{boxed} +} +% then numbering convention +\DeclareOption{part}{ + \renewcommand{\ALG@within}{part} + \setboolean{ALG@within}{true} +} +\DeclareOption{chapter}{ + \renewcommand{\ALG@within}{chapter} + \setboolean{ALG@within}{true} +} +\DeclareOption{section}{ + \renewcommand{\ALG@within}{section} + \setboolean{ALG@within}{true} +} +\DeclareOption{subsection}{ + \renewcommand{\ALG@within}{subsection} + \setboolean{ALG@within}{true} +} +\DeclareOption{subsubsection}{ + \renewcommand{\ALG@within}{subsubsection} + \setboolean{ALG@within}{true} +} +\DeclareOption{nothing}{ + \renewcommand{\ALG@within}{nothing} + \setboolean{ALG@within}{true} +} +\DeclareOption*{\edef\ALG@name{\CurrentOption}} + +% ALGORITHM +% +\ProcessOptions +\floatstyle{\ALG@floatstyle} +\ifthenelse{\boolean{ALG@within}}{ + \ifthenelse{\equal{\ALG@within}{part}} + {\newfloat{algorithm}{htbp}{loa}[part]}{} + \ifthenelse{\equal{\ALG@within}{chapter}} + {\newfloat{algorithm}{htbp}{loa}[chapter]}{} + \ifthenelse{\equal{\ALG@within}{section}} + {\newfloat{algorithm}{htbp}{loa}[section]}{} + \ifthenelse{\equal{\ALG@within}{subsection}} + {\newfloat{algorithm}{htbp}{loa}[subsection]}{} + \ifthenelse{\equal{\ALG@within}{subsubsection}} + {\newfloat{algorithm}{htbp}{loa}[subsubsection]}{} + \ifthenelse{\equal{\ALG@within}{nothing}} + {\newfloat{algorithm}{htbp}{loa}}{} +}{ + \newfloat{algorithm}{htbp}{loa} +} +\floatname{algorithm}{\ALG@name} + +\newcommand{\listofalgorithms}{\listof{algorithm}{\listalgorithmname}} + diff --git a/branches/pisi-1.1/doc/algorithmic.sty b/branches/pisi-1.1/doc/algorithmic.sty new file mode 100644 index 00000000..68f18aa0 --- /dev/null +++ b/branches/pisi-1.1/doc/algorithmic.sty @@ -0,0 +1,158 @@ +% ALGORITHMIC STYLE -- Released 8 APRIL 1996 +% for LaTeX version 2e +% Copyright -- 1994 Peter Williams +% E-mail PeterWilliams@dsto.defence.gov.au +\NeedsTeXFormat{LaTeX2e} +\ProvidesPackage{algorithmic} +\typeout{Document Style `algorithmic' - environment} +% +\RequirePackage{ifthen} +\RequirePackage{calc} +\newboolean{ALC@noend} +\setboolean{ALC@noend}{false} +\newcounter{ALC@line} +\newcounter{ALC@rem} +\newlength{\ALC@tlm} +% +\DeclareOption{noend}{\setboolean{ALC@noend}{true}} +% +\ProcessOptions +% +% ALGORITHMIC +\newcommand{\algorithmicrequire}{\textbf{Require:}} +\newcommand{\algorithmicensure}{\textbf{Ensure:}} +\newcommand{\algorithmiccomment}[1]{\{#1\}} +\newcommand{\algorithmicend}{\textbf{end}} +\newcommand{\algorithmicif}{\textbf{if}} +\newcommand{\algorithmicthen}{\textbf{then}} +\newcommand{\algorithmicelse}{\textbf{else}} +\newcommand{\algorithmicelsif}{\algorithmicelse\ \algorithmicif} +\newcommand{\algorithmicendif}{\algorithmicend\ \algorithmicif} +\newcommand{\algorithmicfor}{\textbf{for}} +\newcommand{\algorithmicforall}{\textbf{for all}} +\newcommand{\algorithmicdo}{\textbf{do}} +\newcommand{\algorithmicendfor}{\algorithmicend\ \algorithmicfor} +\newcommand{\algorithmicwhile}{\textbf{while}} +\newcommand{\algorithmicendwhile}{\algorithmicend\ \algorithmicwhile} +\newcommand{\algorithmicloop}{\textbf{loop}} +\newcommand{\algorithmicendloop}{\algorithmicend\ \algorithmicloop} +\newcommand{\algorithmicrepeat}{\textbf{repeat}} +\newcommand{\algorithmicuntil}{\textbf{until}} +\def\ALC@item[#1]{% +\if@noparitem \@donoparitem + \else \if@inlabel \indent \par \fi + \ifhmode \unskip\unskip \par \fi + \if@newlist \if@nobreak \@nbitem \else + \addpenalty\@beginparpenalty + \addvspace\@topsep \addvspace{-\parskip}\fi + \else \addpenalty\@itempenalty \addvspace\itemsep + \fi + \global\@inlabeltrue +\fi +\everypar{\global\@minipagefalse\global\@newlistfalse + \if@inlabel\global\@inlabelfalse \hskip -\parindent \box\@labels + \penalty\z@ \fi + \everypar{}}\global\@nobreakfalse +\if@noitemarg \@noitemargfalse \if@nmbrlist \refstepcounter{\@listctr}\fi \fi +\sbox\@tempboxa{\makelabel{#1}}% +\global\setbox\@labels + \hbox{\unhbox\@labels \hskip \itemindent + \hskip -\labelwidth \hskip -\ALC@tlm + \ifdim \wd\@tempboxa >\labelwidth + \box\@tempboxa + \else \hbox to\labelwidth {\unhbox\@tempboxa}\fi + \hskip \ALC@tlm}\ignorespaces} +% +\newenvironment{algorithmic}[1][0]{ +\let\@item\ALC@item + \newcommand{\ALC@lno}{% +\ifthenelse{\equal{\arabic{ALC@rem}}{0}} +{{\footnotesize \arabic{ALC@line}:}}{}% +} +\let\@listii\@listi +\let\@listiii\@listi +\let\@listiv\@listi +\let\@listv\@listi +\let\@listvi\@listi +\let\@listvii\@listi + \newenvironment{ALC@g}{ + \begin{list}{\ALC@lno}{ \itemsep\z@ \itemindent\z@ + \listparindent\z@ \rightmargin\z@ + \topsep\z@ \partopsep\z@ \parskip\z@\parsep\z@ + \leftmargin 1em + \addtolength{\ALC@tlm}{\leftmargin} + } + } + {\end{list}} + \newcommand{\ALC@it}{\addtocounter{ALC@line}{1}\addtocounter{ALC@rem}{1}\ifthenelse{\equal{\arabic{ALC@rem}}{#1}}{\setcounter{ALC@rem}{0}}{}\item} + \newcommand{\ALC@com}[1]{\ifthenelse{\equal{##1}{default}}% +{}{\ \algorithmiccomment{##1}}} + \newcommand{\REQUIRE}{\item[\algorithmicrequire]} + \newcommand{\ENSURE}{\item[\algorithmicensure]} + \newcommand{\STATE}{\ALC@it} + \newcommand{\COMMENT}[1]{\algorithmiccomment{##1}} + \newenvironment{ALC@if}{\begin{ALC@g}}{\end{ALC@g}} + \newenvironment{ALC@for}{\begin{ALC@g}}{\end{ALC@g}} + \newenvironment{ALC@whl}{\begin{ALC@g}}{\end{ALC@g}} + \newenvironment{ALC@loop}{\begin{ALC@g}}{\end{ALC@g}} + \newenvironment{ALC@rpt}{\begin{ALC@g}}{\end{ALC@g}} + \renewcommand{\\}{\@centercr} + \newcommand{\IF}[2][default]{\ALC@it\algorithmicif\ ##2\ \algorithmicthen% +\ALC@com{##1}\begin{ALC@if}} + \newcommand{\ELSE}[1][default]{\end{ALC@if}\ALC@it\algorithmicelse% +\ALC@com{##1}\begin{ALC@if}} + \newcommand{\ELSIF}[2][default]% +{\end{ALC@if}\ALC@it\algorithmicelsif\ ##2\ \algorithmicthen% +\ALC@com{##1}\begin{ALC@if}} + \newcommand{\FOR}[2][default]{\ALC@it\algorithmicfor\ ##2\ \algorithmicdo% +\ALC@com{##1}\begin{ALC@for}} + \newcommand{\FORALL}[2][default]{\ALC@it\algorithmicforall\ ##2\ % +\algorithmicdo% +\ALC@com{##1}\begin{ALC@for}} + \newcommand{\WHILE}[2][default]{\ALC@it\algorithmicwhile\ ##2\ % +\algorithmicdo% +\ALC@com{##1}\begin{ALC@whl}} + \newcommand{\LOOP}[1][default]{\ALC@it\algorithmicloop% +\ALC@com{##1}\begin{ALC@loop}} + \newcommand{\REPEAT}[1][default]{\ALC@it\algorithmicrepeat% +\ALC@com{##1}\begin{ALC@rpt}} + \newcommand{\UNTIL}[1]{\end{ALC@rpt}\ALC@it\algorithmicuntil\ ##1} + \ifthenelse{\boolean{ALC@noend}}{ + \newcommand{\ENDIF}{\end{ALC@if}} + \newcommand{\ENDFOR}{\end{ALC@for}} + \newcommand{\ENDWHILE}{\end{ALC@whl}} + \newcommand{\ENDLOOP}{\end{ALC@loop}} + }{ + \newcommand{\ENDIF}{\end{ALC@if}\ALC@it\algorithmicendif} + \newcommand{\ENDFOR}{\end{ALC@for}\ALC@it\algorithmicendfor} + \newcommand{\ENDWHILE}{\end{ALC@whl}\ALC@it\algorithmicendwhile} + \newcommand{\ENDLOOP}{\end{ALC@loop}\ALC@it\algorithmicendloop} + } + \renewcommand{\@toodeep}{} + \begin{list}{\ALC@lno}{\setcounter{ALC@line}{0}\setcounter{ALC@rem}{0}% + \itemsep\z@ \itemindent\z@ \listparindent\z@% + \partopsep\z@ \parskip\z@ \parsep\z@% + \labelsep 0.5em \topsep 0.2em% +\ifthenelse{\equal{#1}{0}} + {\labelwidth 0.5em } + {\labelwidth 1.2em } +\leftmargin\labelwidth \addtolength{\leftmargin}{\labelsep} + \ALC@tlm\labelsep + } +} +{\end{list}} + + + + + + + + + + + + + + + diff --git a/branches/pisi-1.1/doc/component-notes.txt b/branches/pisi-1.1/doc/component-notes.txt new file mode 100644 index 00000000..10d9be84 --- /dev/null +++ b/branches/pisi-1.1/doc/component-notes.txt @@ -0,0 +1,167 @@ +component'larla ilgili notlar + +1. Temel feature'lar (Eray) +-------------------- + +Iki yerde component tag'i tanimladik simdiye kadar, bir +source'larda bir de binary'lerde. Binary'de tanimlanan component default +olarak source'daki tanimi inherit ediyor ve tanimlanan component'i override +edebiliyor. + +Bir component temel olarak bir meta-package, icerisinde paketler olan +bir paket. Bir component'in icerisinde bir takim source'lar ve bir takim +binary'ler bulunuyor diye bakabiliriz. Sanirim query'leri bu sekilde yapmak +mumkun olmali. + +$ pisi list-components +$ pisi info system.base + Source packages in system.base: + .... + .... + Binary packages in system.base: + .... + .... + +gibi ozellikler eklemeyi umit ediyorum. + +Source'larin component tag'leri de, gene sadece bir senaryo konusuyorum, +direkt olarak directory structure'indan inherit alinacak. Ayni zamanda o pspec +icin de bir tane component tanimlananack default olarak, ve bu component o +scope'da tanimlanmis olan butun binary package'lari icerecek. + +Ornegin diyelim ki a/b/c/pspec.xml var ve c1 c2 c3 seklinde uc tane paket +tanimliyor. Hic bir component tanimi yapilmadigi zaman otomatik olarak bir +a.b.c component'i olusturulacak, ve bu component'in icerisinde c1 c2 c3 +bulunacak. + +$ pisi info a.b.c + Source packages: + c - oldur beni yarim sen olmazsan biterim + Binary packages: + c1 - bu aksam demlenmemek sonum olur benim + c2 - sincaplarla konustum butun gece + c3 - her gul gordugumde icim kan aglar + +Bu varsayilan davranis, ama bunu degistirmek mumkun olacak. Burada yamuk +gozukebilecek bir sey var, o da tek bir paket oldugunda sanki biraz +redundancy olmasi, o takdirde bir optimizasyon olarak, diyelim ki +a/b/c/pspec.xml'in icerisinde tek bir paket tanimli c1 + +$ pisi info a.b + .... + c1 - dil dil dillerdeyim + +olabilir bu durumda, bu genel agac mantiginda bir sorun yaratmayacaktir. + + +2. Temel tanım (Barış) + +fiziksel aitlik: kdebase'den çıkan kcontrol gibi; grup aitliği: pdf +göstericileri gibi... + + +3. Database XML ayrımı (Barış) + +Component database'i ile component.xml ayrı olmalı. Component database, +pisi'nin pisi-index.xml dosyasını okuyarak oluşturacağı bir veritabanı. Hangi +paketler hangi componentlere dahil, vs. sorguları bu veritabanından +yapılacak. + +Pspec dosyasını hatırlayalım. İçerisinde bir diye bir tag var. +Oluşturulacak paketin hangi component'e ait olduğunu belirtiyor. Bu bilgi +pisi-index.xml dosyasına da koyulmalı. + +4. PL modüllerine benzerlik (Eray) + +component tag'leri Java ya da python'daki gibi +directory yapisindan cikiyor. Yani pisi bir programlama dili olsaydi +component'lar module'ler ya da package'larla es anlamli olacakti. + +5. Mereology ve Eray'ın açıklama çabaları (Eray) + +PartOf iliskisi hakkında su kadarini soylemek yeter: eger a b'nin bir parcasiysa, +a'nin fonksiyonu b'nin fonksiyonunun bir parcasidir. Bu da fiziksel sistemler +icin bir principle of compositionality'nin varligindan hareket eder [*] + +Genel olarak da software engineering ve AI camiasinda module'un tanimi gayet +iyi bilinir. Bir modul icerisindeki bagimliliklar yogundur. Moduller arasindaki +bagimliliklar zayiftir. + +Bu tanim sadece software engineering'de degil, nesneler arasindaki +benzerliklerin incelendigi bir cok disiplinde kullanilan informal bir tanim, +ama tabii ki formule dokulmus bir ton hali var.Sırf bu tanımı taban alarak +yazılmış başarılı kümeleme (clustering) algoritmaları var. + +Modulerlik tanimi verilen *fiziksel* modul ve parcasi olma iliskisiyle +birlesince birlikte install edilip remove edilme yahut ortak bağımlılıklara +sahip olma tanımlarına götürür. + +Bu tanımı analiz edebilmek için parcasi olma iliskisininin anlamini +korumamiz yeterli. Temel olarak + +  kol insanin parcasidir + +iliskisi burada yer aliyor.  Eger + +  a partof b + +turu iliskilerde a paket ya da component, ve b component ise, a ve b'nin +iliskisinin kol ve insan iliskisi gibi olmasini bekleriz. Eger bu parca-butun +iliskisini ihlal ediyorsa o zaman muhtemelen yanlis bir iliski bulunmus +demektir. + +Bunun turlu sonuclari da onem sirasina gore soyle dizilebilir: + +1. Fonksiyonlarin bolunebilmesi prensibinden: (basit bir sonuç) +   a's function is part-of b's function + + orneğin kolun fonksiyonu insanın total fonksiyonunun bir parçasıdır. + +paket ornegi: pisi'nin fonksiyonu olan paket yükleme/çıkarma system.base'in fonksiyonunun, yani temel pardus sisteminin fonksiyonunun bir parçasıdır. + +2. Karmasik sistemlerde birbirine dayanan ufak parcalarin kararli yapilar +meydana getirmesi prensibinden: (evrimsel sonuç) + +   if a is a part-of c, and b is a part-of c, then it follows that a and b +may: +     a. have many interdependencies +     b. share in their origins +  (which are about the same thing) + +2b. sonucunun bizim durumumuza uygulanması, source code'un aynı kaynaktan +çıkması, birlikte inşa edilmeleri gibi şartları getirir. Bunun oldukça +olasi, en azindan insa metodlarinin ve kaynaklarının birbirine benzemesini +bekleriz. Yalniz, 2.a'daki bagimliliklar sadece insa ile degil ayni zamanda +calisma ile de alakalidir. Kaynakları farklı parçaların birbirine bağımlı +hale gelebilecegini unutmamak gerekir. + +Fedora'nin yaklasiminin hos tarafi, boyle teknik ayrintilara girmeden "package +group" mantigini kullanmasiydi, ama bizim belli bir anlami olan parca-butun +iliskisini korumamiz daha mantikli bir yazilim ontology'si ortaya +cikaracaktir. Onlarin yaklasimi ise "anything goes", grup ile kategori'nin +temel bir farki yok cunku, herhangi bir bakis acisi olabilir "grup". + + +6. Modulleri test etmek (!!) (Eray) +------------------------------------ + +PISI'deki seçilen implementation detaylari bir tanima cok fazla commitment +yapmiyor, implementation'ın getirdiği tek şey paketleri bir ağaca koymak. +Component'ların seçiminin ne kadar kaliteli olduğunu belirleyemiyor. +Paketlerin ve componentlarin secimi yapiyor, ki kritik olan o. +Yalnız modulerlik tanimindan ve part-of iliskisinin +  if a is part of b, and b is part of c, then a is part of c. +  if a is part of b, and b is not part of c, then a is not part of c. +gibi sonuclari getirmesinden hareketle (ki bunlar klasik computational +ontology) kismen test edebilecegimiz bir sekil aliyor ornegin bagimlilik +graph'ini cluster ederek, ya da her modul icin bir modulerlik sayisi +hesaplayarak. Daha az formal olarak da bu sonuçları kafamızda yürüterek +yaptığımız componentların ne kadar akla yatkın olduğunu bulabiliriz. + + +7. Cagların frugalware önerisi +------------------------------ + +http://ftp.frugalware.org/pub/frugalware/frugalware-current/source/ +adresindeki yerleşimin hem source hem de binary depo için uygulanmasını ve +kategori, componentların da bundan çıkartılmasını öneriyorum. diff --git a/branches/pisi-1.1/doc/dependency.pdf b/branches/pisi-1.1/doc/dependency.pdf new file mode 100644 index 00000000..7f9dd07f Binary files /dev/null and b/branches/pisi-1.1/doc/dependency.pdf differ diff --git a/branches/pisi-1.1/doc/dependency.tex b/branches/pisi-1.1/doc/dependency.tex new file mode 100644 index 00000000..0cff823e --- /dev/null +++ b/branches/pisi-1.1/doc/dependency.tex @@ -0,0 +1,381 @@ +%%% Local Variables: +%%% mode: latex +%%% TeX-master: t +%%% End: + +\documentclass[a4paper,11pt]{article} +\usepackage{graphicx} +\usepackage{algorithm} +\usepackage{algorithmic} +\usepackage{amsmath} +\usepackage{amstext} +\usepackage{amsfonts} +\usepackage{amsbsy} +\usepackage{amsthm} +\usepackage{prettyref} +%\newrefformat{alg}{Algorithm~\ref{#1}} +%\newrefformat{eq}{Equation~\ref{#1}} +%\newrefformat{lem}{Lemma~\ref{#1}} +%\newrefformat{thm}{Theorem~\ref{#1}} +%\newrefformat{chp}{Chapter~\ref{#1}} +%\newrefformat{sec}{Section~\ref{#1}} +%\newrefformat{apx}{Appendix~\ref{#1}} +%\newrefformat{tab}{Table~\ref{#1}} +%\newrefformat{fig}{Figure~\ref{#1}} +%usepackage[active]{srcltx} +\title{ Dependency Resolution in PISI} + +\author{Eray \"{O}zkural} + +\date{\today} + +\begin{document} + +\maketitle + +\section{Introduction} + +Dependency resolution in package management systems have a +significance in that they are the key to providing system stability +and internet upgrades. The scale of package databases requires the +dependency resolution mechanism to be efficient and correct, +motivating a closer look at the theory. + + +\section{Review} + +Dependency resolution has been taken in the most general setting as +the famous SAT problem of propositional logic. If we consider a system +$D$ of dependency statements $D_i$, each statement can be taken as a +proposition in propositional logic which states, for instance: + +$D_i$: if package $a$ is installed or package $b$ is installed, then +package $i$ is installable.\\ +... + +The system is thus understood as the conjunction of such facts, giving +us a logical programming formulation to determine installation +conditions. Note that for simplicity we do not consider the nuances in +upgrade and remove operations at the moment. + +However, using a SAT solver for this operation may be shooting a fly +with a bazooka. We observe that only certain forms of propositions +will be necessary for a dependency system. Furthermore, as we shall +see further constraints and optimizations may be required of the +system that are not modelled well with the SAT problem. + +We use a graph theoretic approach instead. A directed graph (digraph) +$G=(V,E)$ is formally a set of vertices $V$ and a set of edges $E$ +where each edge $(u,v)$ represents an edge from a vertex to another. +Accessor functions $V(G)$ and $E(G)$ yield the vertex and edge set of +the graph $G$. Topological sort of a graph gives a total ordering of +the vertices in which there are only forward edges. A vertex induced +subgraph of $G$ by vertex set $A$ contains only the vertex set $A$ and +edges incident to members of $A$. + +\section{Package operation planning} + +The dependency resolution problem may be viewed as a simple forward +chaining problem, where we would like to begin from an initial state +$S_0$ and by following allowable system transitions $t_i: S \to S$, +arrive at a desired system state $S_f$ (where $S$ is the set of all states). + +A system state $S_i$ is defined as the set of installed packages on +the system together with their versions, i.e. $S_i = \{ (x,v) : x +\text{ is installed}, v=version(x)\} $. An atomic system transition +$t_i$ chains one system state into another, making one ACID change on +the system. The usual atomic transitions are the single package +install, remove and reinstall (upgrade or downgrade) operations found +in low-level package management code of PISI. Note that in PISI, an +upgrade operation is identical to a remove operation followed by an +install operation (which sets it apart from some other packaging +systems). + +A package operation plan is thus naturally conceived of as a sequence +of atomic system transitions. Given an initial state and a final +state, the job of the package operation planner is to determine +whether there is a plan, and if so find the "best" one. + +Where there are no versions involved (e.g. upgrade/downgrade), we will +replace the pair $(x,v)$ with $x$ in the definitions for simplicity. + +\subsection{System consistency} + +It is worth mentioning here the concept of system consistency. As in a +database transaction, it is not acceptable that the system violates an +invariant afterwards. In the context of PISI, system consistency is +composed of two conditions for the current set of installed packages. +\begin{enumerate} +\item All package dependencies are satisfied (we may call this a + closed system) +\item No package conflicts are present. +\end{enumerate} + +Therefore, by atomic transition we also mean one that does not corrupt +system consistency. The system is thus never in an inconsistent +state. We will explain the conflicts later, for the present let us +look at the dependency condition. + +\subsection{Solving the simplest case with topological sorting} + +We will now concentrate on a simple form of the problem which can be +solved with topological sorting. This form is not concerned with +versions. Neither do we consider remote repositories. From an initial +set of packages $S_0$, we would like to install in addition a new set +$A$ of packages obtaining $S_f = S_0 \cup A$, for a static set of package +relations. + +The only relations considered are of the form: $a$ Depends on $b$, or +more briefly $aDb$. The graph of all such simple dependency relations +is a digraph $G$. For each dependency relation $aDb$, there is an +edge $a \to b$ in $G$. Accessing graph $G$ usually requires a database +operation and is therefore expensive. + +We now consider the digraph $G_A$ of the minimal set of simple +dependency relations which contains all information required to +construct a plan to install packages $A$. $G_A$ is a vertex induced +graph such that the fringe of $A$, e.g. vertices with out-degree $0$ +depend only on packages that are already installed (or none). Vertices +of $G_A$ are taken from $S_f$. First, let us explain the labelling +scheme. Already installed vertices are labelled with 'i'. Packages to +be added are labelled with 'a', and packages to be installed due to +dependencies are labelled with 'd'. We construct the graph as follows +\begin{algorithm} + \caption{$\textsc{Make-}G_A(G, A)$} + \label{alg:cons-graph} + \begin{algorithmic}[1] +\STATE $G_A \gets$ vertex induced subgraph of $G$ by $A$ labelled with 'a' +\REPEAT + \STATE done $\gets$ true + \FOR{each $u \in V(G_A)$ with out-degree $0$} + \FOR{ $v \in adj(u) $ of $G$} + \IF{$v \notin V(G_A)$} + \STATE done $\gets$ false + \IF{$v$ is installed} + \STATE label $v$ with 'i' + \ELSE + \STATE label $v$ with 'd' + \ENDIF + \STATE add $(u,v)$ to $G_A$ + \ENDIF + \ENDFOR + \ENDFOR +\UNTIL{done} +\end{algorithmic} +\end{algorithm} + +By this iterative expansion, we do a minimum number of database +accesses to $G$ and construct a dependency graph in memory. If the +$G_A$'s fringe has vertices with non 'i'-labels, then $A$ cannot be +installed. Otherwise, we find a topological sort $L$ of $G_A$, and in +the reverse order, install packages for vertices labelled with +'a' or 'd'. Observe that, by definition of a topological sort, +installing packages in the reverse order of a topological sort +guarantees that no package is installed before all of its dependencies +are installed. Thus, this yields a consistency-preserving plan. + +\subsection{Dependency conditions} + +In the PISI specification, we allow a dependency to specify a local +condition, for instance a program may require a dependency on +\texttt{libx} with pardus source release $3$ or greater. Another +program may require a dependency on a particular source release. These +conditions are local because they can be computed over the elements of +system state $S_i$, e.g. package (name, version) pairs. Let us denote this +condition by a predicate $P(b)$ such that $aDb$ iff $P(b)$. The +predicate $P$ for the dependency $aDb$ can be stored as edge data for +$(u,v)$ on the graph. +% thus when we say $P(u,v)$, this means the predicate stored on +%$(u,v)$ edge for vertex $v$. + +In this case, the vertices of the package dependency graph $G$ and the +planning graph $G_A$ retain the version information along with the +package name. The dependency relation thus holds between two pairs +$(p_1,v_1)$ and $(p_2,v_2)$, satisfying a given predicate +$P(p_2,v_2)$. When constructing the graph, we therefore take this +predicate into account and admit a new edge $(u,v)$, and thus a new +vertex $v$ into $G_A$ if and only if the target vertex satisfies +$P(v)$. + +\subsection{Conflicts and COMAR dependencies} + +The tags \texttt{Conflicts} and \texttt{Provides} in PISI are +inherited from Debian distribution. A conflict between two packages +($a$ conflicts with $b$) is a symmetric relation that prevents the +packages $a,b$ from being installed simultaneously (It is +sufficient that only one direction of the relation is declared, the +other direction is inferred). Provision in the form of $a$ provides $A$ +denotes that $a$ implements a virtual package abstraction $A$. + +In PISI, a package can provide an object of a COMAR Object Model (OM), +and is currently the only model of a ``virtual package''. In the +following example, let $a_1,a_2,\ldots,a_n$ provide the OM $A$. A package +can depend on another package's OM, for instance $b$ comar-depends on +$A$ (or in short form $bDA$) (Currently, conditions on virtual +dependencies are not supported). In this case, it is sufficient that +only one of the $a_i$ are installed. To resolve this, the user is +asked to choose from a list of alternatives immediately, since +otherwise there is unavoidable combinatorial explosion (in the form of +having to consider $\Pi_{bDA}num(A)$ graphs in the worst case where +$num(A)$ is the number of alternatives for comar OM $A$; the problem +is that there seems to be no simple solution to solve satisfiability +with arbitrary disjunctions in package dependency, short of a $SAT$ +solver). + +The resolution of conflicts to maintain system consistency condition +$2$ is easier to achieve. This can be satisfied by disallowing +installation of a package that would violate the condition, or +removing currently installed packages which conflict with the newly +installed package and its dependencies. In most package managers, the +second option is confirmed by the user for making it easier. In the +install operation, after constructing the partial dependency graph +$G_A$, we merely have to check whether any conflict appears among the +vertices of $G_A$. If so, then the operation is untenable, since $G_A$ +shows the future state of the installed system. Since a conflict is +symmetric, it is represented as a bidirectional edge $a \leftrightarrow b$. To +distinguish dependencies from conflicts, the edges would have to be +labelled in this case, for instance with 'd' and 'c'. The removal +option can be implemented by invoking a multi-package remove operation +on the packages in conflict. + +\subsection{Remove operation} + +Dependency resolution for remove operation is similar to install. The +only significant difference is that we remove the packages in the +topological order rather than installing packages in the reverse +topological order. + +\section{Remote repositories and upgrade operation} + + +The upgrade operation is more complicated. First of all, the system +has to distinguish between the current relation graph (e.g. +dependencies and conflicts), and the future relation graph which may +be different in rather important aspects. In theory, we allow any +dependency and conflict to change. Therefore, we have a $G_0$ which +represents the current relations (among installed packages) in the +system, and a $G_f$ which is probably taken from a remote package +repository. We begin by noting that $G_0$ and $G_f$ have to be +compatible. That is, to say, if a package $(p_1,v_1)$ is shared across +two graphs, then the declarations made by the package are one and the +same. + +$G_0$ can be calculated from the package information (e.g. metadata) +of the installed packages and is stored by PISI in a dedicated +database. $G_f$ is most likely constructed from a PISI Index file +corresponding to a particular package repository. Accessing both of +these entities is expensive and we should take care to minimize access +as in the previous section. + +To preserve consistency during individual transitions, the planner can +choose to remove a minimum number of packages from the system to bring +it to a clean state, and then install the new versions of these +packages in the correct order. Let us assume that it is indeed +possible to achieve this ``clean state''. Apparently, this is not +always possible because other packages may depend on the package(s) to +upgrade. At any rate, to achieve this, first we need to +calculate subgraphs of $G_0$ and $G_f$. We can calculate alternative +plans from these subgraphs if need be. + +Let $A$ be the set of packages to be upgraded from a given repository. +$G_{A,0}$ is the subgraph of $G_{0}$ induced by the ``upgrade +closure'' of $A$. The ``upgrade closure'' of a set $A$ of packages is +defined as a minimal set of packages $B \supseteq A$ such that there is no +package in $B$ that requires an upgrade for $A$ to be upgraded. This +is found by assuming that the current system state $S_0$ is +consistent, and by constructing a relation graph of the future state +of the system to detect the dependencies that have changed. + +Obviously, to make a plan, we must first know the goal state. In a +multi-package upgrade, the exact details of the goal state depend on +the graph $G_f$ of the repository. Thus, we construct a graph +$G_{A,f}$ that is a vertex-induced subgraph of $G_f$ such that it +contains all information relevant to upgrading packages $A$. We begin +by a vertex induced subgraph of $G_f$ by $A$. These are the packages +that will be upgraded in any case. Then, we make a pass on the +vertices, and look at all the outgoing edges, we compare whether this +edge has changed in any substantial way from the previous version. In +particular, we are interested in whether the predicate of the edge is +valid for the version of the same package in our current system. Every +compared vertex in this manner is marked done, and the edges not valid +for the current system pull new unmarked vertices into $G_f$, this +continues until there are no unmarked vertices left. Hence, the +vertices of $G_f$ are the packages that must be upgraded. + +To actually carry out the upgrade a strategy is to upgrade all the +packages in $G_{A,f}$ in some order. A good order is again the reverse +topological order order, in fact, the upgrade operation is merely a +special case of a multi-package installation code that can install +from a remote repository, since a multi-package installation can +contain upgrades in addition to new packages. However, in case no +package depends on the packages to be upgraded, then we can carry out +a completely consistency-preserving plan as discussed above. The +conflicts are resolved in the usual fashion, by removing those +packages in conflict with new packages that are installed. This can be +accomplished by invoking a remove operation prior to the upgrade +operation. + +\section{Examples} + +\subsection{A single package upgrade} + +goal: upgrade $(a,1)$ to $(a,2)$\\ +\\ +rules:\\ + $(a,1)$ depends on $(b,1), (c,1)$ \\ + $(a,1)$ conflicts with $(d,1)$\\ + $(a,2)$ depends on $(c,3), (d,2)$\\ + $(a,2)$ conflicts with $(b,1)$\\ +\\ +initial state:\\ + $(a,1), (b,1), (c,1)$ installed \\ + +In this case, we can find a consistency-preserving plan in terms of +install and remove operations. +\\ +plan:\\ + remove $(a,1)$\\ + remove $(b,1)$\\ + remove $(c,1)$\\ + install $(c,3)$\\ + install $(d,2)$\\ + install $(a,2)$\\ + +\subsection{Another upgrade} + +goal: upgrade $(b,1) \to (b,2)$\\ +\\ +current dep: $(a,1) \to[=1] (b,1) \to[=1] (c,1) \to[=1] (d,1)$\\ +repo dep: \nobreakspace{} \nobreakspace$(a,1) \to[=2] (b,2) \to[=2] (c,2) \to[=1] (d,1)$\\ + +In this case, we cannot remove $(b,1)$ because it's locked in the +chain. In fact, here there is no consistency-preserving plan in terms +of atomic single package transitions: install, remove, upgrade. In +these cases, it seems best to resort to upgrade in place, and in the +reverse topological order of dependencies. +\\ +plan:\\ + upgrade $(c,1) \to (c,2)$\\ + upgrade $(b,1) \to (b,2)$\\ + + +\subsection{A multi package remove} + +goal: remove $(a,2), (b,3), (c,2)$\\ +\\ +rules:\\ + $(c,2)$ depends $(a,2)$\\ + $(d,2)$ depends on $(b,3), (c,2)$\\ + $(e,1)$ depends on $(a,2)$\\ + $(f,2)$ depends on $(e,1)$\\ + $(g,2)$ depends on $(e,1)$\\\\ +plan:\\ + remove $(f,2)$\\ + remove $(g,2)$\\ + remove $(e,1)$\\ + remove $(d,2)$\\ + remove $(b,3)$\\ + remove $(c,2)$\\ + remove $(a,2)$\\ +\end{document} + diff --git a/branches/pisi-1.1/doc/introduction-to-pisi.odp b/branches/pisi-1.1/doc/introduction-to-pisi.odp new file mode 100644 index 00000000..66be0b52 Binary files /dev/null and b/branches/pisi-1.1/doc/introduction-to-pisi.odp differ diff --git a/branches/pisi-1.1/doc/package_versions.pdf b/branches/pisi-1.1/doc/package_versions.pdf new file mode 100644 index 00000000..09228904 Binary files /dev/null and b/branches/pisi-1.1/doc/package_versions.pdf differ diff --git a/branches/pisi-1.1/doc/package_versions.tex b/branches/pisi-1.1/doc/package_versions.tex new file mode 100644 index 00000000..620e10a1 --- /dev/null +++ b/branches/pisi-1.1/doc/package_versions.tex @@ -0,0 +1,187 @@ +\documentclass[a4paper,11pt]{article} + +\title{P\.IS\.I Packages: Version Policy v0.2} +\date{\today} +\author{Eray \"Ozkural and T. Bar\i{}\c s Metin} + + +\begin{document} +\maketitle + +\section*{Revision History} +\begin{itemize} +\item v0.1: Bar\i\c s Metin wrote the first version preparing the outline, +detailed Source Version Section, and started the Section on Release Number. +\item v0.2: Eray \"Ozkural wrote a detailed introduction, added + explanations of release and build numbers, reorganized a bit. +\end{itemize} + +\section{Introduction} + +This document explains the \emph{version policy} that applies to +P\.IS\.I packages. Classically, the issue of distinguishing source and +binary distributions unambiguously has not received a rigorous +treatment in the context of LINUX distributions. We have identified +several shortcomings of the usual practices of extending the original +version with suffixes and prefixes, colorfully illustrated in the +following common problems. + +\begin{description} + +\item[The problem of future downgrades] +The distribution chooses to use a previous version of the package in +the next release. There is no way to indicate this, so ad-hoc +solutions such as version prefixes are used. It is +impossible to denote a future dependency that requires at least this +distribution source release in this case, either. + +\item[The problem of redundant distributions] +A trivial patch has been applied to the source. While few binary +packages have been affected by this change, all binary packages +built from the source are redistributed. + +\item[The problem of underdetermined rebuilds] +There have been rapid changes in the system, and although no +changes have been made to the package source, a new binary +distribution must be prepared. + +\end{description} + +We have devised a slightly new approach in order to alleviate these +problems. Our solution consists of encoding the history of source and +binary package developments in separate version strings we call release and +build numbers. + +Since the source version is usually used by the users and developers +to identify software, we retain the notion of a source version in +P\.IS\.I as a convenience. + +In the following sections, we explain the components of our +versioning scheme. + +\subsection{Source Version} + +Source version is the version number provided by the +upstream maintainer of the source archive used in package. It must +always be the same as the upstream version used. + +\textbf{Example}: If the upstream archive name is +\emph{bash-3.0.tar.gz} the version number of the package is \emph{3.0} + +\subsubsection{Version Suffixes} + +There is a pre-defined list of suffixes a package version can +take. + +\begin{itemize} + \item \textbf{alpha} Source/Package is in alpha state + \item \textbf{beta} Source/Package is in beta state + \item \textbf{pre} Source/Pacgage passed the beta state but stable + version is not relased yet. + \item \textbf{rc} Source/Package is a release-candidate. + \item \textbf{m} Source/Package is a milestone before stable version. + \item \textbf{p} Source/Package is released and some patches are + applied after the release. This is the patch level. +\end{itemize} + +The suffix should be written after the special separator +character \textbf{\_}. And there must allways be a number after a +suffix. \textbf{Example}: packagename-1.0\_beta1 + +The basic order of the priorities for suffixes is:\newline +\emph{p $>$ (no suffix) $>$ m $>$ rc $>$ pre $>$ beta $>$ alpha}. + +The scope of a source version string is global in the literal +sense. It shall not vary from repository to repository. + +The support for these special suffixes as well as usual alphanumeric +version string ordering has been implemented in P\.IS\.I. + +\section{Identifying Package Sources} + +A P\.IS\.I source has three identity elements written under +\texttt{SOURCE} tag: name, source version, and source release number. +We usually say just version and release number/release instead of +source version and source release number, respectively. Name is available in +the \texttt{} tag. Version and release are available in the last +\texttt{} element of \texttt{} tag of a \texttt{PSPEC}. + +The name of a source package is constant throughout its revision +history. The version is the original version, given by its +programmers. Release is a positive integer. Name and release +is sufficient to uniquely identify a particular PISI source revision. +That is, version and release are independent. + +\subsection{Release Number} + +Release number is the number of the changes that are made to the +package source since the initial version in the distribution source. A +change can be a patch applied to the source archive, modification in +the actions.py, pspec.xml or any file in the source package +directory. This change is indicated in \texttt{} tags manually +by the package maintainer. + +The initial release of a package is by default \texttt{1}. The release +number always increments by $1$ in each revision in the +\texttt{History}, even the slightest ones, but it never decrements. + +The scope of the release number is a given distribution, regardless of +its version, e.g. Pardus. + +In the future, PISI will have strict checks for release numbers. + +\subsection{Dependency Specifications} + +We allow a package to use both source version and release to identify +a particular version or a range of package versions. + +\section{Identifying Binary packages} + +A PISI binary package is produced from a PISI source package. It has a +name that is constant throughout the history of the source package, +and it inherits the source version and release number from the source +package. However, a binary package has in addition a binary build +number. Shortly, build number or just build. For each of the +architecture targets, e.g. particular binaries, it also has an +architecture tag. + +A binary package is uniquely identified by its name, build number, and +architecture regardless of the source version. + +\section{Build Number} + +Similarly to source release number, binary build number is the number +of changes that are made to a binary package. By change, we mean any +bit change. The existence of a change is tested by comparing the +cryptographic checksums in files.xml with those of the previous build, and the +build number is automatically determined by the P\.IS\.I build system. +The build number starts from $1$ as in release number, and increments +by one with each binary change. + +The user never interferes with the build number himself. However, if +the user fails to provide the previous build, then a package without +a build number is built. A package without a build number is evaluated +on the basis of release number, which is guaranteed to exist. + +The scope of a build number is a given distribution build environment +for a particular architecture, which may vary from repository to +repository. Therefore, it is not used in dependency +specifications. However, the system does assume that a build of a +given package and architecture is unique in a given repository. + +\section{Package File Names} + +A P\.IS\.I binary package file name contains all the components relevant +to its identification, separated by dashes: +\begin{verbatim} + ---.pisi +\end{verbatim} + +\section{Future Work} + +In the future, it may be necessary to extend the notion of release +number and build number to support branches and forks of a +distribution. A proposal was to have CVS-like branching, but it +was dismissed as unnecessary. + +\end{document} diff --git a/branches/pisi-1.1/doc/pisi-db.xmi b/branches/pisi-1.1/doc/pisi-db.xmi new file mode 100644 index 00000000..6966ed90 --- /dev/null +++ b/branches/pisi-1.1/doc/pisi-db.xmi @@ -0,0 +1,532 @@ + + + + + umbrello uml modeller http://uml.sf.net + 1.5.4 + UnicodeUTF8 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/branches/pisi-1.1/doc/prettyref.sty b/branches/pisi-1.1/doc/prettyref.sty new file mode 100644 index 00000000..67940f3b --- /dev/null +++ b/branches/pisi-1.1/doc/prettyref.sty @@ -0,0 +1,37 @@ +%% +%% This is file `prettyref.sty', +%% generated with the docstrip utility. +%% +%% The original source files were: +%% +%% prettyref.dtx (with options: `style') +%% +%% Copyright (c) 1995 Kevin Ruland +%% +%% +%% prettyref v3.0 +%% +%% Copyright 1995,1998. by Kevin Ruland kevin@rodin.wustl.edu +%% +\ProvidesPackage{prettyref}[1998/07/09 v3.0] +\def\newrefformat#1#2{% + \@namedef{pr@#1}##1{#2}} +\newrefformat{eq}{\textup{(\ref{#1})}} +\newrefformat{lem}{Lemma \ref{#1}} +\newrefformat{thm}{Theorem \ref{#1}} +\newrefformat{cha}{Chapter \ref{#1}} +\newrefformat{sec}{Section \ref{#1}} +\newrefformat{tab}{Table \ref{#1} on page \pageref{#1}} +\newrefformat{fig}{Figure \ref{#1} on page \pageref{#1}} +\def\prettyref#1{\@prettyref#1:} +\def\@prettyref#1:#2:{% + \expandafter\ifx\csname pr@#1\endcsname\relax% + \PackageWarning{prettyref}{Reference format #1\space undefined}% + \ref{#1:#2}% + \else% + \csname pr@#1\endcsname{#1:#2}% + \fi% +} +\endinput +%% +%% End of file `prettyref.sty'. diff --git a/branches/pisi-1.1/pisi-cli b/branches/pisi-1.1/pisi-cli new file mode 100755 index 00000000..0dab058c --- /dev/null +++ b/branches/pisi-1.1/pisi-cli @@ -0,0 +1,89 @@ +#!/usr/bin/python +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. +# + +import sys +import locale +import traceback +import exceptions +import signal +import bsddb3.db as db + +import pisi.ui +import pisi.context as ctx +import pisi.cli.pisicli as pisicli + +import gettext +gettext.bindtextdomain('pisi', "/usr/share/locale") +gettext.textdomain('pisi') +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +def sig_handler(sig, frame): + if sig == signal.SIGTERM: + exit() + +def exit(): + try: + pisi.api.finalize() + except KeyboardInterrupt: # raised pending interrupt + pass + sys.exit(1) + +def handle_exception(exception, value, tb): + signal.signal(signal.SIGINT, signal.SIG_IGN) # disable further interrupts + ui = pisi.cli.CLI() # make a temporary UI + show_traceback = False + if exception == exceptions.KeyboardInterrupt: + ui.error(_("Keyboard Interrupt: Exiting...")) + exit() + elif isinstance(value, pisi.Error): + ui.error(_("Program Terminated.")) + show_traceback = ctx.get_option('debug') + elif isinstance(value, db.DBRunRecoveryError): + ui.error(_("""A database operation has been aborted. You should run pisi +again for normal DB recovery procedure. Make sure you have free disk space. +You have to run rebuild-db only when there is file corruption and database upgrades.""")) + elif isinstance(value, pisi.Exception): + show_traceback = True + ui.error(_("""Unhandled internal exception. +Please file a bug report. (http://bugs.uludag.org.tr)""")) + else: + # For any other exception (possibly Python exceptions) show + # the traceback! + show_traceback = ctx.get_option('debug') + ui.error(_("System Error. Program Terminated.")) + + if ctx.get_option('debug'): + ui.error(u"%s: %s" % (exception, value)) + else: + ui.error(unicode(value)) + + ui.info(_("Please use 'pisi help' for general help.")) + + if show_traceback: + ui.info(_("Traceback:")) + traceback.print_tb(tb) + else: + if not exception is pisicli.Error: + ui.info(_("Use --debug to see a traceback.")) + + exit() + +if __name__ == "__main__": + + sys.excepthook = handle_exception + + signal.signal(signal.SIGTERM, sig_handler) + + locale.setlocale(locale.LC_ALL, '') + cli = pisicli.PisiCLI() + cli.run_command() diff --git a/branches/pisi-1.1/pisi-cli2.5 b/branches/pisi-1.1/pisi-cli2.5 new file mode 100755 index 00000000..aaeea9d1 --- /dev/null +++ b/branches/pisi-1.1/pisi-cli2.5 @@ -0,0 +1,78 @@ +#!/usr/bin/python2.5 +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. +# + +import sys +import locale +import traceback +import exceptions +import signal +import bsddb3.db as db + +import pisi.ui +import pisi.context as ctx +import pisi.cli.pisicli as pisicli + +import gettext +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +def exit(): + pisi.api.finalize() + sys.exit(1) + +def handle_exception(exception, value, tb): + signal.signal(signal.SIGINT, signal.SIG_IGN) # disable further interrupts + ui = pisi.cli.CLI() # make a temporary UI + show_traceback = False + if exception == exceptions.KeyboardInterrupt: + ui.error(_("Keyboard Interrupt: Exiting...")) + exit() + elif isinstance(value, pisi.Error): + ui.error(_("Program Terminated.")) + show_traceback = ctx.get_option('debug') + elif isinstance(value, db.DBRunRecoveryError): + ui.error(_("""A database operation has been aborted. You should run pisi +again for normal DB recovery procedure. Make sure you have free disk space. +You have to run rebuild-db only when there is file corruption and database upgrades.""")) + elif isinstance(value, pisi.Exception): + show_traceback = True + ui.error(_("""Unhandled internal exception. +Please file a bug report. (http://bugs.uludag.org.tr)""")) + else: + # For any other exception (possibly Python exceptions) show + # the traceback! + show_traceback = ctx.get_option('debug') + ui.error(_("System Error. Program Terminated.")) + + if ctx.get_option('debug'): + ui.error(u"%s: %s" % (exception, value)) + else: + ui.error(unicode(value)) + + ui.info(_("Please use 'pisi help' for general help.")) + + if show_traceback: + ui.info(_("Traceback:")) + traceback.print_tb(tb) + else: + if not exception is pisicli.Error: + ui.info(_("Use --debug to see a traceback.")) + + exit() + +if __name__ == "__main__": + + sys.excepthook = handle_exception + + locale.setlocale(locale.LC_ALL, '') + cli = pisicli.PisiCLI() + cli.run_command() diff --git a/branches/pisi-1.1/pisi-spec.dtd b/branches/pisi-1.1/pisi-spec.dtd new file mode 100644 index 00000000..95bef334 --- /dev/null +++ b/branches/pisi-1.1/pisi-spec.dtd @@ -0,0 +1,123 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/branches/pisi-1.1/pisi/__init__.py b/branches/pisi-1.1/pisi/__init__.py new file mode 100644 index 00000000..1845ae5f --- /dev/null +++ b/branches/pisi-1.1/pisi/__init__.py @@ -0,0 +1,40 @@ +# -*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. +# + +# PiSi version + +__version__ = "1.1.7" + +__dbversion__ = "1.1.5" +__filesdbversion__ = "1.0.5" # yes, this is the real bottleneck + +__all__ = [ 'api', 'config', 'configfile', 'packagedb', 'installdb', 'search' ] + +# FIXME: Exception shadows builtin Exception. This is no good. +class Exception(Exception): + """Class of exceptions that must be caught and handled within PiSi""" + def __str__(self): + s = u'' + for x in self.args: + if s != '': + s += '\n' + s += unicode(x) + return s + +class Error(Exception): + """Class of exceptions that lead to program termination""" + pass + +import pisi.api + +# FIXME: can't do this due to name clashes in config and other singletons booo +#pisi.api import * diff --git a/branches/pisi-1.1/pisi/actionsapi/__init__.py b/branches/pisi-1.1/pisi/actionsapi/__init__.py new file mode 100644 index 00000000..362ca288 --- /dev/null +++ b/branches/pisi-1.1/pisi/actionsapi/__init__.py @@ -0,0 +1,27 @@ +# -*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. +# + +import pisi + +class Error(pisi.Error): + pass + +class Exception(pisi.Exception): + pass + +import pisi.context as ctx + +def error(msg): + if ctx.config.get_option('ignore_action_errors'): + ctx.ui.error(msg) + else: + raise Error(msg) diff --git a/branches/pisi-1.1/pisi/actionsapi/autotools.py b/branches/pisi-1.1/pisi/actionsapi/autotools.py new file mode 100644 index 00000000..8b7ec62e --- /dev/null +++ b/branches/pisi-1.1/pisi/actionsapi/autotools.py @@ -0,0 +1,164 @@ +# -*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. + +# Standard Python Modules +import os + +import gettext +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +# Pisi Modules +import pisi.context as ctx + +# ActionsAPI Modules +import pisi.actionsapi +import pisi.actionsapi.get as get +from pisi.actionsapi.shelltools import system +from pisi.actionsapi.shelltools import can_access_file +from pisi.actionsapi.shelltools import unlink +from pisi.actionsapi.libtools import gnuconfig_update + +class ConfigureError(pisi.actionsapi.Error): + def __init__(self, value=''): + pisi.actionsapi.Error.__init__(self, value) + self.value = value + ctx.ui.error(value) + if can_access_file('config.log'): + ctx.ui.error(_('Please attach the config.log to your bug report:\n%s/config.log') % os.getcwd()) + +class MakeError(pisi.actionsapi.Error): + def __init__(self, value=''): + pisi.actionsapi.Error.__init__(self, value) + self.value = value + ctx.ui.error(value) + +class InstallError(pisi.actionsapi.Error): + def __init__(self, value=''): + pisi.actionsapi.Error.__init__(self, value) + self.value = value + ctx.ui.error(value) + +class RunTimeError(pisi.actionsapi.Error): + def __init__(self, value=''): + pisi.actionsapi.Error.__init__(self, value) + self.value = value + ctx.ui.error(value) + +def configure(parameters = ''): + '''configure source with given parameters = "--with-nls --with-libusb --with-something-usefull"''' + if can_access_file('configure'): + gnuconfig_update() + + args = './configure \ + --prefix=/%s \ + --build=%s \ + --mandir=/%s \ + --infodir=/%s \ + --datadir=/%s \ + --sysconfdir=/%s \ + --localstatedir=/%s \ + %s' % (get.defaultprefixDIR(), \ + get.HOST(), get.manDIR(), \ + get.infoDIR(), get.dataDIR(), \ + get.confDIR(), get.localstateDIR(), parameters) + + if system(args): + raise ConfigureError(_('Configure failed.')) + else: + raise ConfigureError(_('No configure script found.')) + +def rawConfigure(parameters = ''): + '''configure source with given parameters = "--prefix=/usr --libdir=/usr/lib --with-nls"''' + if can_access_file('configure'): + gnuconfig_update() + + if system('./configure %s' % parameters): + raise ConfigureError(_('Configure failed.')) + else: + raise ConfigureError(_('No configure script found.')) + +def compile(parameters = ''): + #FIXME: Only one package uses this until now, hmmm + system('%s %s %s' % (get.CC(), get.CFLAGS(), parameters)) + +def make(parameters = ''): + '''make source with given parameters = "all" || "doc" etc.''' + if system('make %s %s' % (get.makeJOBS(), parameters)): + raise MakeError(_('Make failed.')) + +def fixInfoDir(): + infoDir = "%s/usr/share/info/dir" % get.installDIR() + if can_access_file(infoDir): + unlink(infoDir) + +def install(parameters = '', argument = 'install'): + '''install source into install directory with given parameters''' + if can_access_file('makefile') or can_access_file('Makefile') or can_access_file('GNUmakefile'): + args = 'make prefix=%(prefix)s/%(defaultprefix)s \ + datadir=%(prefix)s/%(data)s \ + infodir=%(prefix)s/%(info)s \ + localstatedir=%(prefix)s/%(localstate)s \ + mandir=%(prefix)s/%(man)s \ + sysconfdir=%(prefix)s/%(conf)s \ + %(parameters)s \ + %(argument)s' % {'prefix': get.installDIR(), + 'defaultprefix': get.defaultprefixDIR(), + 'man': get.manDIR(), + 'info': get.infoDIR(), + 'localstate': get.localstateDIR(), + 'conf': get.confDIR(), + 'data': get.dataDIR(), + 'parameters': parameters, + 'argument':argument} + + if system(args): + raise InstallError(_('Install failed.')) + else: + fixInfoDir() + else: + raise InstallError(_('No Makefile found.')) + + +def rawInstall(parameters = '', argument = 'install'): + '''install source into install directory with given parameters = PREFIX=%s % get.installDIR()''' + if can_access_file('makefile') or can_access_file('Makefile') or can_access_file('GNUmakefile'): + if system('make %s %s' % (parameters, argument)): + raise InstallError(_('Install failed.')) + else: + fixInfoDir() + else: + raise InstallError(_('No Makefile found.')) + +def aclocal(parameters = ''): + '''generates an aclocal.m4 based on the contents of configure.in.''' + if system('aclocal %s' % parameters): + raise RunTimeError(_('Running aclocal failed.')) + +def autoconf(parameters = ''): + '''generates a configure script''' + if system('autoconf %s' % parameters): + raise RunTimeError(_('Running autoconf failed.')) + +def autoreconf(parameters = ''): + '''re-generates a configure script''' + if system('autoreconf %s' % parameters): + raise RunTimeError(_('Running autoreconf failed.')) + +def automake(parameters = ''): + '''generates a makefile''' + if system('automake %s' % parameters): + raise RunTimeError(_('Running automake failed.')) + +def autoheader(parameters = ''): + '''generates templates for configure''' + if system('autoheader %s' % parameters): + raise RunTimeError(_('Running autoheader failed.')) diff --git a/branches/pisi-1.1/pisi/actionsapi/cmaketools.py b/branches/pisi-1.1/pisi/actionsapi/cmaketools.py new file mode 100644 index 00000000..375bf492 --- /dev/null +++ b/branches/pisi-1.1/pisi/actionsapi/cmaketools.py @@ -0,0 +1,114 @@ +# -*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. + +# Standard Python Modules +import os + +import gettext +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +# Pisi Modules +import pisi.context as ctx +from pisi.util import join_path + +# ActionsAPI Modules +import pisi.actionsapi +import pisi.actionsapi.get as get +from pisi.actionsapi.shelltools import system +from pisi.actionsapi.shelltools import can_access_file +from pisi.actionsapi.shelltools import unlink + +class ConfigureError(pisi.actionsapi.Error): + def __init__(self, value=''): + pisi.actionsapi.Error.__init__(self, value) + self.value = value + ctx.ui.error(value) + if can_access_file('config.log'): + ctx.ui.error(_('Please attach the config.log to your bug report:\n%s/config.log') % os.getcwd()) + +class MakeError(pisi.actionsapi.Error): + def __init__(self, value=''): + pisi.actionsapi.Error.__init__(self, value) + self.value = value + ctx.ui.error(value) + +class InstallError(pisi.actionsapi.Error): + def __init__(self, value=''): + pisi.actionsapi.Error.__init__(self, value) + self.value = value + ctx.ui.error(value) + +class RunTimeError(pisi.actionsapi.Error): + def __init__(self, value=''): + pisi.actionsapi.Error.__init__(self, value) + self.value = value + ctx.ui.error(value) + +def configure(parameters = '', installPrefix = '/%s' % get.defaultprefixDIR(), sourceDir = '.'): + '''configure source with given cmake parameters = "-DCMAKE_BUILD_TYPE -DCMAKE_CXX_FLAGS ... "''' + if can_access_file(join_path(sourceDir, 'CMakeLists.txt')): + args = 'cmake -DCMAKE_INSTALL_PREFIX=%s %s %s' % (installPrefix, parameters, sourceDir) + + if system(args): + raise ConfigureError(_('Configure failed.')) + else: + raise ConfigureError(_('No configure script found for cmake.')) + +def make(parameters = ''): + '''build source with given parameters''' + if can_access_file('Makefile'): + if system('make %s %s' % (get.makeJOBS(), parameters)): + raise MakeError(_('Make failed.')) + else: + raise InstallError(_('No Makefile found.')) + +def fixInfoDir(): + infoDir = "%s/usr/share/info/dir" % get.installDIR() + if can_access_file(infoDir): + unlink(infoDir) + +def install(parameters = '', argument = 'install'): + '''install source into install directory with given parameters''' + if can_access_file('makefile') or can_access_file('Makefile') or can_access_file('GNUmakefile'): + args = 'make prefix=%(prefix)s/%(defaultprefix)s \ + datadir=%(prefix)s/%(data)s \ + infodir=%(prefix)s/%(info)s \ + localstatedir=%(prefix)s/%(localstate)s \ + mandir=%(prefix)s/%(man)s \ + sysconfdir=%(prefix)s/%(conf)s \ + %(parameters)s \ + %(argument)s' % {'prefix': get.installDIR(), + 'defaultprefix': get.defaultprefixDIR(), + 'man': get.manDIR(), + 'info': get.infoDIR(), + 'localstate': get.localstateDIR(), + 'conf': get.confDIR(), + 'data': get.dataDIR(), + 'parameters': parameters, + 'argument':argument} + + if system(args): + raise InstallError(_('Install failed.')) + else: + fixInfoDir() + else: + raise InstallError(_('No Makefile found.')) + +def rawInstall(parameters = '', argument = 'install'): + '''install source into install directory with given parameters = PREFIX=%s % get.installDIR()''' + if can_access_file('makefile') or can_access_file('Makefile') or can_access_file('GNUmakefile'): + if system('make %s %s' % (parameters, argument)): + raise InstallError(_('Install failed.')) + else: + fixInfoDir() + else: + raise InstallError(_('No Makefile found.')) diff --git a/branches/pisi-1.1/pisi/actionsapi/coreutils.py b/branches/pisi-1.1/pisi/actionsapi/coreutils.py new file mode 100644 index 00000000..381e80bf --- /dev/null +++ b/branches/pisi-1.1/pisi/actionsapi/coreutils.py @@ -0,0 +1,80 @@ +#-*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. + +# Standard Python Modules +import re +import sys +from itertools import izip +from itertools import imap +from itertools import count +from itertools import ifilter +from itertools import ifilterfalse + +# ActionsAPI +import pisi.actionsapi + +def cat(filename): + return file(filename).xreadlines() + +class grep: + '''keep only lines that match the regexp''' + def __init__(self, pat, flags = 0): + self.fun = re.compile(pat, flags).match + def __ror__(self, input): + return ifilter(self.fun, input) + +class tr: + '''apply arbitrary transform to each sequence element''' + def __init__(self, transform): + self.tr = transform + def __ror__(self, input): + return imap(self.tr, input) + +class printto: + '''print sequence elements one per line''' + def __init__(self, out = sys.stdout): + self.out = out + def __ror__(self,input): + for line in input: + print >> self.out, line + +printlines = printto(sys.stdout) + +class terminator: + def __init__(self,method): + self.process = method + def __ror__(self,input): + return self.process(input) + +aslist = terminator(list) +asdict = terminator(dict) +astuple = terminator(tuple) +join = terminator(''.join) +enum = terminator(enumerate) + +class sort: + def __ror__(self,input): + ll = list(input) + ll.sort() + return ll +sort = sort() + +class uniq: + def __ror__(self,input): + for i in input: + try: + if i == prev: + continue + except NameError: + pass + prev = i + yield i +uniq = uniq() diff --git a/branches/pisi-1.1/pisi/actionsapi/get.py b/branches/pisi-1.1/pisi/actionsapi/get.py new file mode 100644 index 00000000..a9b54322 --- /dev/null +++ b/branches/pisi-1.1/pisi/actionsapi/get.py @@ -0,0 +1,193 @@ +# -*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. + +# Standart Python Modules +import os +import sys + +import gettext +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +# PiSi Modules +import pisi.actionsapi +import pisi.context as ctx + +# ActionsAPI Modules +import pisi.actionsapi.variables + +class BinutilsError(pisi.actionsapi.Error): + def __init__(self, value=''): + pisi.actionsapi.Error.__init__(self, value) + self.value = value + ctx.ui.error(value) + +# Globals +env = pisi.actionsapi.variables.glb.env +dirs = pisi.actionsapi.variables.glb.dirs + +def curDIR(): + '''returns current work directory's path''' + return os.getcwd() + +def curKERNEL(): + '''returns currently running kernel's version''' + return os.uname()[2] + +def curPYTHON(): + ''' returns currently used python's version''' + (a, b, c, x, y) = sys.version_info + return 'python%s.%s' % (a, b) + +def curPERL(): + ''' returns currently used perl's version''' + return os.path.realpath("/usr/bin/perl").split("perl")[1] + +def ENV(environ): + '''returns any given environ variable''' + try: + return os.environ[environ]; + except KeyError: + return None + +# PİSİ Related Functions + +def pkgDIR(): + '''returns the path of binary packages''' + '''Default: /var/cache/pisi/packages''' + return env.pkg_dir + +def workDIR(): + return env.work_dir + +def installDIR(): + '''returns the path of binary packages''' + return env.install_dir + +# PSPEC Related Functions + +def srcNAME(): + return env.src_name + +def srcVERSION(): + return env.src_version + +def srcRELEASE(): + return env.src_release + +def srcTAG(): + return '%s-%s-%s' % (env.src_name, env.src_version, env.src_release) + +def srcDIR(): + return '%s-%s' % (env.src_name, env.src_version) + +# Build Related Functions + +def HOST(): + return env.host + +def CHOST(): + # FIXME: Currently it behave same as HOST, + # but will be used for cross-compiling when PİSİ ready... + return env.host + +def CFLAGS(): + return env.cflags + +def CXXFLAGS(): + return env.cxxflags + +def LDFLAGS(): + return env.ldflags + +def makeJOBS(): + return env.jobs + +# Directory Related Functions + +def docDIR(): + return dirs.doc + +def sbinDIR(): + return dirs.sbin + +def infoDIR(): + return dirs.info + +def manDIR(): + return dirs.man + +def dataDIR(): + return dirs.data + +def confDIR(): + return dirs.conf + +def localstateDIR(): + return dirs.localstate + +def defaultprefixDIR(): + return dirs.defaultprefix + +def kdeDIR(): + return dirs.kde + +def qtDIR(): + return dirs.qt + +# Binutils Variables + +def existBinary(bin): + # determine if path has binary + path = os.environ['PATH'].split(':') + for directory in path: + if os.path.exists(os.path.join(directory, bin)): + return True + return False + +def getBinutilsInfo(util): + cross_build_name = '%s-%s' % (HOST(), util) + if not existBinary(cross_build_name): + if not existBinary(util): + raise BinutilsError(_('Util %s cannot be found') % util) + else: + ctx.ui.debug(_('Warning: %s does not exist, using plain name %s') \ + % (cross_build_name, util)) + return util + else: + return cross_build_name + +def AR(): + return getBinutilsInfo('ar') + +def AS(): + return getBinutilsInfo('as') + +def CC(): + return getBinutilsInfo('gcc') + +def CXX(): + return getBinutilsInfo('g++') + +def LD(): + return getBinutilsInfo('ld') + +def NM(): + return getBinutilsInfo('nm') + +def RANLIB(): + return getBinutilsInfo('ranlib') + +def F77(): + return getBinutilsInfo('g77') + +def GCJ(): + return getBinutilsInfo('gcj') diff --git a/branches/pisi-1.1/pisi/actionsapi/kde.py b/branches/pisi-1.1/pisi/actionsapi/kde.py new file mode 100644 index 00000000..9a58b515 --- /dev/null +++ b/branches/pisi-1.1/pisi/actionsapi/kde.py @@ -0,0 +1,91 @@ +# -*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. + +# standard python modules +import os + +import gettext +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +# Pisi Modules +import pisi.context as ctx + +# ActionsAPI Modules +import pisi.actionsapi +import pisi.actionsapi.get as get +from pisi.actionsapi.shelltools import system +from pisi.actionsapi.pisitoolsfunctions import * +from pisi.actionsapi.shelltools import can_access_file + +class ConfigureError(pisi.actionsapi.Error): + def __init__(self, value=''): + pisi.actionsapi.Error.__init__(self, value) + self.value = value + ctx.ui.error(value) + if can_access_file('config.log'): + ctx.ui.error(_('\n!!! Please attach the config.log to your bug report:\n%s/config.log') % os.getcwd()) + +class MakeError(pisi.actionsapi.Error): + def __init__(self, value=''): + pisi.actionsapi.Error.__init__(self, value) + self.value = value + ctx.ui.error(value) + +class InstallError(pisi.actionsapi.Error): + def __init__(self, value=''): + pisi.actionsapi.Error.__init__(self, value) + self.value = value + ctx.ui.error(value) + +def configure(parameters = ''): + ''' parameters = '--with-nls --with-libusb --with-something-usefull ''' + if can_access_file('configure'): + args = './configure \ + --prefix=%s \ + --build=%s \ + --with-x \ + --enable-mitshm \ + --with-xinerama \ + --with-qt-dir=%s \ + --enable-mt \ + --with-qt-libraries=%s/lib \ + --disable-dependency-tracking \ + --disable-debug \ + %s' % (get.kdeDIR(), get.HOST(), get.qtDIR(), get.qtDIR(), parameters) + + if system(args): + raise ConfigureError(_('Configure failed.')) + else: + raise ConfigureError(_('No configure script found.')) + +def make(parameters = ''): + '''make source with given parameters = "all" || "doc" etc.''' + if system('make %s %s' % (get.makeJOBS(), parameters)): + raise MakeError(_('Make failed.')) + +def install(parameters = 'install'): + if can_access_file('Makefile'): + args = 'make DESTDIR=%s destdir=%s %s' % (get.installDIR(), get.installDIR(), parameters) + + if system(args): + raise InstallError(_('Install failed.')) + else: + raise InstallError(_('No Makefile found.')) + +def domo(sourceFile, locale, destinationFile ): + '''inserts the mo files in the list of files into /usr/kde/VERSION/share/locale/LOCALE/LC_MESSAGES''' + + '''example call: kde.domo("po/tr.po", "tr", "pam_login.mo")''' + + system('msgfmt %s' % sourceFile) + makedirs(join_path(get.installDIR(), get.kdeDIR(), 'share/locale', locale, 'LC_MESSAGES')) + move('messages.mo', join_path(get.installDIR(), get.kdeDIR(), 'share/locale', locale, 'LC_MESSAGES', destinationFile)) diff --git a/branches/pisi-1.1/pisi/actionsapi/libtools.py b/branches/pisi-1.1/pisi/actionsapi/libtools.py new file mode 100644 index 00000000..a70fe2e8 --- /dev/null +++ b/branches/pisi-1.1/pisi/actionsapi/libtools.py @@ -0,0 +1,72 @@ +# -*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. + +# Standard Python Modules +import os + +import gettext +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +# Pisi-Core Modules +import pisi.context as ctx +from pisi.util import join_path + +# ActionsAPI Modules +import pisi.actionsapi +from pisi.actionsapi.shelltools import * +import pisi.actionsapi.get as get + +class RunTimeError(pisi.actionsapi.Error): + def __init__(self, value=''): + pisi.actionsapi.Error.__init__(self, value) + self.value = value + ctx.ui.error(value) + +def preplib(sourceDirectory = '/usr/lib'): + sourceDirectory = join_path(get.installDIR(), sourceDirectory) + if can_access_directory(sourceDirectory): + if system('/sbin/ldconfig -n -N %s' % sourceDirectory): + raise RunTimeError(_('Running ldconfig failed.')) + +def gnuconfig_update(): + ''' copy newest config.* onto source\'s ''' + for root, dirs, files in os.walk(os.getcwd()): + for fileName in files: + if fileName in ['config.sub', 'config.guess']: + targetFile = os.path.join(root, fileName) + if os.path.islink(targetFile): + unlink(targetFile) + copy('/usr/share/gnuconfig/%s' % fileName, join_path(root, fileName)) + ctx.ui.info(_('GNU Config Update Finished.')) + +def libtoolize(parameters = ''): + if system('/usr/bin/libtoolize %s' % parameters): + raise RunTimeError(_('Running libtoolize failed.')) + +def gen_usr_ldscript(dynamicLib): + + makedirs('%s/usr/lib' % get.installDIR()) + + destinationFile = open('%s/usr/lib/%s' % (get.installDIR(), dynamicLib), 'w') + content = ''' +/* GNU ld script + Since Pardus has critical dynamic libraries + in /lib, and the static versions in /usr/lib, + we need to have a "fake" dynamic lib in /usr/lib, + otherwise we run into linking problems. +*/ +GROUP ( /lib/%s ) +''' % dynamicLib + + destinationFile.write(content) + destinationFile.close() + chmod('%s/usr/lib/%s' % (get.installDIR(), dynamicLib)) diff --git a/branches/pisi-1.1/pisi/actionsapi/perlmodules.py b/branches/pisi-1.1/pisi/actionsapi/perlmodules.py new file mode 100644 index 00000000..e4f4483a --- /dev/null +++ b/branches/pisi-1.1/pisi/actionsapi/perlmodules.py @@ -0,0 +1,84 @@ +# -*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. + +# standard python modules +import os +import glob + +import gettext +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +# Pisi Modules +import pisi.context as ctx + +# ActionsAPI Modules +import pisi.actionsapi +import pisi.actionsapi.get as get +from pisi.actionsapi.shelltools import system +from pisi.actionsapi.shelltools import can_access_file +from pisi.actionsapi.shelltools import export +from pisi.actionsapi.shelltools import unlink + +class ConfigureError(pisi.actionsapi.Error): + def __init__(self, value=''): + pisi.actionsapi.Error.__init__(self, value) + self.value = value + ctx.ui.error(value) + +class MakeError(pisi.actionsapi.Error): + def __init__(self, value=''): + pisi.actionsapi.Error.__init__(self, value) + self.value = value + ctx.ui.error(value) + +class InstallError(pisi.actionsapi.Error): + def __init__(self, value=''): + pisi.actionsapi.Error.__init__(self, value) + self.value = value + ctx.ui.error(value) + +def configure(parameters = ''): + '''configure source with given parameters.''' + export('PERL_MM_USE_DEFAULT', '1') + if can_access_file('Build.PL'): + if system('perl Build.PL installdirs=vendor destdir=%s' % get.installDIR()): + raise ConfigureError, _('Configure failed.') + else: + if system('perl Makefile.PL %s PREFIX=/usr INSTALLDIRS=vendor DESTDIR=%s' % (parameters, get.installDIR())): + raise ConfigureError, _('Configure failed.') + +def make(parameters = ''): + '''make source with given parameters.''' + if can_access_file('Makefile'): + if system('make %s' % parameters): + raise MakeError, _('Make failed.') + else: + if system('perl Build build'): + raise MakeError, _('perl build failed.') + +def install(parameters = 'install'): + '''install source with given parameters.''' + if can_access_file('Makefile'): + if system('make %s' % parameters): + raise InstallError, _('Make failed.') + else: + if system('perl Build install'): + raise MakeError, _('perl install failed.') + + fixLocalPod() + +def fixLocalPod(): + podFiles = glob.glob("%s/usr/lib/*/*/*/perllocal.pod" % get.installDIR()) + + for podFile in podFiles: + if can_access_file(podFile): + unlink(podFile) diff --git a/branches/pisi-1.1/pisi/actionsapi/pisitools.py b/branches/pisi-1.1/pisi/actionsapi/pisitools.py new file mode 100644 index 00000000..b82f51f7 --- /dev/null +++ b/branches/pisi-1.1/pisi/actionsapi/pisitools.py @@ -0,0 +1,236 @@ +#-*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. + +'''supports globs in sourceFile arguments''' + + +# Standart Python Modules +import os +import glob +import sys +import fileinput +import re + +import gettext +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +# Pisi Modules +import pisi.context as ctx +from pisi.util import join_path, remove_prefix + +# ActionsAPI Modules +import pisi.actionsapi +import pisi.actionsapi.get as get +from pisi.actionsapi.pisitoolsfunctions import * +from pisi.actionsapi.shelltools import * + +from pisi.actionsapi import error + +def dobin(sourceFile, destinationDirectory = '/usr/bin'): + '''insert a executable file into /bin or /usr/bin''' + ''' example call: pisitools.dobin("bin/xloadimage", "/bin", "xload") ''' + executable_insinto(join_path(get.installDIR(), destinationDirectory), sourceFile) + +def dodir(destinationDirectory): + '''creates a directory tree''' + makedirs(join_path(get.installDIR(), destinationDirectory)) + +def dodoc(*sourceFiles): + '''inserts the files in the list of files into /usr/share/doc/PACKAGE''' + readable_insinto(join_path(get.installDIR(), join_path('/usr/share/doc', get.srcTAG())), *sourceFiles) + +def doexe(sourceFile, destinationDirectory): + '''insert a executable file into destination directory''' + + ''' example call: pisitools.doexe("kde-3.4.sh", "/etc/X11/Sessions")''' + executable_insinto(join_path(get.installDIR(), destinationDirectory), sourceFile) + +def dohtml(*sourceFiles): + '''inserts the files in the list of files into /usr/share/doc/PACKAGE/html''' + + ''' example call: pisitools.dohtml("doc/doxygen/html/*")''' + destionationDirectory = join_path(get.installDIR(), 'usr/share/doc' ,get.srcTAG(), 'html') + + if not can_access_directory(destionationDirectory): + makedirs(destionationDirectory) + + allowed_extensions = ['.png', '.gif', '.html', '.htm', '.jpg', '.css', '.js'] + disallowed_directories = ['CVS'] + + for sourceFile in sourceFiles: + for source in glob.glob(sourceFile): + if os.path.isfile(source) and os.path.splitext(source)[1] in allowed_extensions: + system('install -m0644 "%s" %s' % (source, destionationDirectory)) + if os.path.isdir(source) and os.path.basename(source) not in disallowed_directories: + eraser = os.path.split(source)[0] + for root, dirs, files in os.walk(source): + newRoot = remove_prefix(eraser, root) + for sourcename in files: + if os.path.splitext(sourcename)[1] in allowed_extensions: + makedirs(join_path(destionationDirectory, newRoot)) + system('install -m0644 %s %s' % (join_path(root, sourcename), join_path(destionationDirectory, newRoot, sourcename))) + +def doinfo(*sourceFiles): + '''inserts the into files in the list of files into /usr/share/info''' + readable_insinto(join_path(get.installDIR(), get.infoDIR()), *sourceFiles) + +def dolib(sourceFile, destinationDirectory = '/usr/lib'): + '''insert the library into /usr/lib''' + + '''example call: pisitools.dolib_a("libz.a")''' + '''example call: pisitools.dolib_a("libz.so")''' + sourceFile = join_path(os.getcwd(), sourceFile) + destinationDirectory = join_path(get.installDIR(), destinationDirectory) + + lib_insinto(sourceFile, destinationDirectory, 755) + +def dolib_a(sourceFile, destinationDirectory = '/usr/lib'): + '''insert the static library into /usr/lib with permission 0644''' + + '''example call: pisitools.dolib_a("staticlib/libvga.a")''' + sourceFile = join_path(os.getcwd(), sourceFile) + destinationDirectory = join_path(get.installDIR(), destinationDirectory) + + lib_insinto(sourceFile, destinationDirectory, 644) + +def dolib_so(sourceFile, destinationDirectory = '/usr/lib'): + '''insert the static library into /usr/lib with permission 0755''' + + '''example call: pisitools.dolib_so("pppd/plugins/minconn.so")''' + sourceFile = join_path(os.getcwd(), sourceFile) + destinationDirectory = join_path(get.installDIR(), destinationDirectory) + + lib_insinto(sourceFile, destinationDirectory, 755) + +def doman(*sourceFiles): + '''inserts the man pages in the list of files into /usr/share/man/''' + + '''example call: pisitools.doman("man.1", "pardus.*")''' + manDIR = join_path(get.installDIR(), get.manDIR()) + if not can_access_directory(manDIR): + makedirs(manDIR) + + for sourceFile in sourceFiles: + for source in glob.glob(sourceFile): + try: + pageName, pageDirectory = source[:source.rindex('.')], \ + source[source.rindex('.')+1:] + except ValueError: + error(_('ActionsAPI [doman]: Wrong man page file: %s') % (source)) + + makedirs(join_path(manDIR, '/man%s' % pageDirectory)) + system('install -m0644 %s %s' % (source, join_path(manDIR, '/man%s' % pageDirectory))) + +def domo(sourceFile, locale, destinationFile ): + '''inserts the mo files in the list of files into /usr/share/locale/LOCALE/LC_MESSAGES''' + + '''example call: pisitools.domo("po/tr.po", "tr", "pam_login.mo")''' + + system('msgfmt %s' % sourceFile) + makedirs('%s/usr/share/locale/%s/LC_MESSAGES/' % (get.installDIR(), locale)) + move('messages.mo', '%s/usr/share/locale/%s/LC_MESSAGES/%s' % (get.installDIR(), locale, destinationFile)) + +def domove(sourceFile, destination, destinationFile = ''): + '''moves sourceFile/Directory into destinationFile/Directory''' + + ''' example call: pisitools.domove("/usr/bin/bash", "/bin/bash")''' + ''' example call: pisitools.domove("/usr/bin/", "/usr/sbin")''' + makedirs(join_path(get.installDIR(), destination)) + + for filePath in glob.glob(join_path(get.installDIR(), sourceFile)): + if not destinationFile: + move(filePath, join_path(get.installDIR(), join_path(destination, os.path.basename(filePath)))) + else: + move(filePath, join_path(get.installDIR(), join_path(destination, destinationFile))) + +def rename(sourceFile, destinationFile): + ''' renames sourceFile as destinationFile''' + + ''' example call: pisitools.rename("/usr/bin/bash", "bash.old") ''' + ''' the result of the previous example would be "/usr/bin/bash.old" ''' + + baseDir = os.path.dirname(sourceFile) + + try: + os.rename(join_path(get.installDIR(), sourceFile), join_path(get.installDIR(), baseDir, destinationFile)) + except OSError: + error(_('ActionsAPI [rename]: No such file or directory: %s') % (sourceFile)) + +def dosed(sourceFiles, findPattern, replacePattern = ''): + '''replaces patterns in sourceFiles''' + + ''' example call: pisitools.dosed("/etc/passwd", "caglar", "cem")''' + ''' example call: pisitools.dosed("/etc/passwd", "caglar")''' + ''' example call: pisitools.dosed("/etc/pass*", "caglar")''' + ''' example call: pisitools.dosed("Makefile", "(?m)^(HAVE_PAM=.*)no", r"\1yes")''' + + for sourceFile in glob.glob(sourceFiles): + if can_access_file(sourceFile): + for line in fileinput.input(sourceFile, inplace = 1): + #FIXME: In-place filtering is disabled when standard input is read + line = re.sub(findPattern, replacePattern, line) + sys.stdout.write(line) + else: + raise FileError(_('File does not exist or permission denied: %s') % sourceFile) + +def dosbin(sourceFile, destinationDirectory = '/usr/sbin'): + '''insert a executable file into /sbin or /usr/sbin''' + + ''' example call: pisitools.dobin("bin/xloadimage", "/sbin") ''' + executable_insinto(join_path(get.installDIR(), destinationDirectory), sourceFile) + +def dosym(sourceFile, destinationFile): + '''creates soft link between sourceFile and destinationFile''' + + ''' example call: pisitools.dosym("/usr/bin/bash", "/bin/bash")''' + makedirs(join_path(get.installDIR(), os.path.dirname(destinationFile))) + + try: + os.symlink(sourceFile, join_path(get.installDIR() ,destinationFile)) + except OSError: + error(_('ActionsAPI [dosym]: File exists: %s') % (sourceFile)) + +def insinto(destinationDirectory, sourceFile, destinationFile = '', sym = True): + '''insert a sourceFile into destinationDirectory as a destinationFile with same uid/guid/permissions''' + makedirs(join_path(get.installDIR(), destinationDirectory)) + + if not destinationFile: + for filePath in glob.glob(sourceFile): + if can_access_file(filePath): + copy(filePath, join_path(get.installDIR(), join_path(destinationDirectory, os.path.basename(filePath))), sym) + else: + copy(sourceFile, join_path(get.installDIR(), join_path(destinationDirectory, destinationFile)), sym) + +def newdoc(sourceFile, destinationFile): + '''inserts a sourceFile into /usr/share/doc/PACKAGE/ directory as a destinationFile''' + destinationDirectory = '' #490 + destinationDirectory = os.path.dirname(destinationFile) + destinationFile = os.path.basename(destinationFile) + # Use copy instead of move or let build-install scream like file not found! + copy(sourceFile, destinationFile) + readable_insinto(join_path(get.installDIR(), 'usr/share/doc', get.srcTAG(), destinationDirectory), destinationFile) + +def newman(sourceFile, destinationFile): + '''inserts a sourceFile into /usr/share/man/manPREFIX/ directory as a destinationFile''' + # Use copy instead of move or let build-install scream like file not found! + copy(sourceFile, destinationFile) + doman(destinationFile) + +def remove(sourceFile): + '''removes sourceFile''' + for filePath in glob.glob(join_path(get.installDIR(), sourceFile)): + unlink(filePath) + +def removeDir(destinationDirectory): + '''removes destinationDirectory and its subtrees''' + for directory in glob.glob(join_path(get.installDIR(), destinationDirectory)): + unlinkDir(directory) diff --git a/branches/pisi-1.1/pisi/actionsapi/pisitoolsfunctions.py b/branches/pisi-1.1/pisi/actionsapi/pisitoolsfunctions.py new file mode 100644 index 00000000..d378d866 --- /dev/null +++ b/branches/pisi-1.1/pisi/actionsapi/pisitoolsfunctions.py @@ -0,0 +1,80 @@ +#-*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. + +# Generic functions for common usage of pisitools # + +# Standart Python Modules +import os +import glob + +import gettext +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +# Pisi Modules +import pisi.context as ctx + +# ActionsAPI Modules +import pisi.actionsapi +from pisi.actionsapi.shelltools import * + +class FileError(pisi.actionsapi.Error): + def __init__(self, value=''): + pisi.actionsapi.Error.__init__(self, value) + self.value = value + ctx.ui.error(value) + +class ArgumentError(pisi.actionsapi.Error): + def __init__(self, value=''): + pisi.actionsapi.Error.__init__(self, value) + self.value = value + ctx.ui.error(value) + +def executable_insinto(destinationDirectory, *sourceFiles): + '''insert a executable file into destinationDirectory''' + + if not sourceFiles or not destinationDirectory: + raise ArgumentError(_('Insufficient arguments.')) + + if not can_access_directory(destinationDirectory): + makedirs(destinationDirectory) + + for sourceFile in sourceFiles: + for source in glob.glob(sourceFile): + # FIXME: use an internal install routine for these + system('install -m0755 -o root -g root %s %s' % (source, destinationDirectory)) + +def readable_insinto(destinationDirectory, *sourceFiles): + '''inserts file list into destinationDirectory''' + + if not sourceFiles or not destinationDirectory: + raise ArgumentError(_('Insufficient arguments.')) + + if not can_access_directory(destinationDirectory): + makedirs(destinationDirectory) + + for sourceFile in sourceFiles: + for source in glob.glob(sourceFile): + system('install -m0644 "%s" %s' % (source, destinationDirectory)) + +def lib_insinto(sourceFile, destinationDirectory, permission = 0644): + '''inserts a library fileinto destinationDirectory with given permission''' + + if not sourceFile or not destinationDirectory: + raise ArgumentError(_('Insufficient arguments.')) + + if not can_access_directory(destinationDirectory): + makedirs(destinationDirectory) + + if os.path.islink(sourceFile): + os.symlink(os.path.realpath(sourceFile), os.path.join(destinationDirectory, sourceFile)) + else: + system('install -m%s %s %s' % (permission, sourceFile, destinationDirectory)) diff --git a/branches/pisi-1.1/pisi/actionsapi/pythonmodules.py b/branches/pisi-1.1/pisi/actionsapi/pythonmodules.py new file mode 100644 index 00000000..f50fc95a --- /dev/null +++ b/branches/pisi-1.1/pisi/actionsapi/pythonmodules.py @@ -0,0 +1,74 @@ +# -*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. + +# standard python modules +import os + +import gettext +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +# Pisi Modules +import pisi.context as ctx + +# ActionsAPI Modules +import pisi.actionsapi +import pisi.actionsapi.get as get +from pisi.actionsapi.shelltools import system, can_access_file, unlink +from pisi.actionsapi.pisitools import dodoc + +class CompileError(pisi.actionsapi.Error): + def __init__(self, value=''): + pisi.actionsapi.Error.__init__(self, value) + self.value = value + ctx.ui.error(value) + +class InstallError(pisi.actionsapi.Error): + def __init__(self, value=''): + pisi.actionsapi.Error.__init__(self, value) + self.value = value + ctx.ui.error(value) + +class RunTimeError(pisi.actionsapi.Error): + def __init__(self, value=''): + pisi.actionsapi.Error.__init__(self, value) + self.value = value + ctx.ui.error(value) + +def compile(parameters = ''): + '''compile source with given parameters.''' + if system('python setup.py build %s' % (parameters)): + raise CompileError, _('Make failed.') + +def install(parameters = ''): + '''does python setup.py install''' + if system('python setup.py install --root=%s --no-compile -O0 %s' % (get.installDIR(), parameters)): + raise InstallError, _('Install failed.') + + DDOCS = 'CHANGELOG COPYRIGHT KNOWN_BUGS MAINTAINERS PKG-INFO \ + CONTRIBUTORS LICENSE COPYING* Change* MANIFEST* README*' + + for doc in DDOCS: + if can_access_file(doc): + dodoc(doc) + +def run(parameters = ''): + '''executes parameters with python''' + if system('python %s' % (parameters)): + raise RunTimeError, _('Running %s failed.') % parameters + +def fixCompiledPy(lookInto = "/usr/lib/%s/" % get.curPYTHON()): + ''' cleans *.py[co] from packages ''' + for root, dirs, files in os.walk("%s/%s" % (get.installDIR(),lookInto)): + for compiledFile in files: + if compiledFile.endswith(".pyc") or compiledFile.endswith(".pyo"): + if can_access_file("%s/%s" % (root,compiledFile)): + unlink("%s/%s" % (root,compiledFile)) diff --git a/branches/pisi-1.1/pisi/actionsapi/scons.py b/branches/pisi-1.1/pisi/actionsapi/scons.py new file mode 100644 index 00000000..82df7ae7 --- /dev/null +++ b/branches/pisi-1.1/pisi/actionsapi/scons.py @@ -0,0 +1,43 @@ +# -*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. + + +# Pisi Modules +import pisi.context as ctx + +import gettext +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +# ActionsAPI Modules +import pisi.actionsapi +import pisi.actionsapi.get as get +from pisi.actionsapi.shelltools import system + +class MakeError(pisi.actionsapi.Error): + def __init__(self, value=''): + pisi.actionsapi.Error.__init__(self, value) + self.value = value + ctx.ui.error(value) + +class InstallError(pisi.actionsapi.Error): + def __init__(self, value=''): + pisi.actionsapi.Error.__init__(self, value) + self.value = value + ctx.ui.error(value) + +def make(parameters = ''): + if system('scons %s' % parameters): + raise MakeError(_('Make failed.')) + +def install(parameters = 'install', prefix = get.installDIR(), argument='prefix'): + if system('scons %s=%s %s' % (argument, prefix, parameters)): + raise InstallError(_('Install failed.')) diff --git a/branches/pisi-1.1/pisi/actionsapi/shelltools.py b/branches/pisi-1.1/pisi/actionsapi/shelltools.py new file mode 100644 index 00000000..47784af8 --- /dev/null +++ b/branches/pisi-1.1/pisi/actionsapi/shelltools.py @@ -0,0 +1,224 @@ +#-*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. + +# Standart Python Modules +import os +import glob +import shutil +import string +import pwd +import grp + +import gettext +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +# Pisi Modules +import pisi.context as ctx + +# ActionsAPI Modules +import pisi.actionsapi +import pisi.actionsapi.get + +from pisi.actionsapi import error +from pisi.util import run_logged +from pisi.util import join_path + +def can_access_file(filePath): + '''test the existence of file''' + return os.access(filePath, os.F_OK) + +def can_access_directory(destinationDirectory): + '''test readability, writability and executablility of directory''' + return os.access(destinationDirectory, os.R_OK | os.W_OK | os.X_OK) + +def makedirs(destinationDirectory): + '''recursive directory creation function''' + try: + if not os.access(destinationDirectory, os.F_OK): + os.makedirs(destinationDirectory) + except OSError: + error(_('Cannot create directory %s') % destinationDirectory) + +def echo(destionationFile, content): + try: + f = open(destionationFile, 'a') + f.write('%s\n' % content) + f.close() + except IOError: + error(_('ActionsAPI [echo]: Can\'t append to file %s.') % (destionationFile)) + +def chmod(filePath, mode = 0755): + '''change the mode of filePath to the mode''' + for fileName in glob.glob(filePath): + if can_access_file(fileName): + try: + os.chmod(fileName, mode) + except OSError: + ctx.ui.error(_('ActionsAPI [chmod]: Operation not permitted: %s (mode: %s)') \ + % (fileName, mode)) + else: + ctx.ui.error(_('ActionsAPI [chmod]: File %s doesn\'t exists.') % (fileName)) + +def chown(filePath, uid = "root", gid = "root"): + '''change the owner and group id of filePath to uid and gid''' + if can_access_file(filePath): + try: + os.chown(filePath, pwd.getpwnam(uid)[2], grp.getgrnam(gid)[2]) + except OSError: + ctx.ui.error(_('ActionsAPI [chown]: Operation not permitted: %s (uid: %s, gid: %s)') \ + % (filePath, uid, gid)) + else: + ctx.ui.error(_('ActionsAPI [chown]: File %s doesn\'t exists.') % filePath) + +def sym(source, destination): + '''creates symbolic link''' + try: + os.symlink(source, destination) + except OSError: + ctx.ui.error(_('ActionsAPI [sym]: Permission denied: %s to %s') % (source, destination)) + +def unlink(filePath): + '''remove the file path''' + if isFile(filePath) or isLink(filePath): + try: + os.unlink(filePath) + except OSError: + ctx.ui.error(_('ActionsAPI [unlink]: Permission denied: %s.') % (filePath)) + elif isDirectory(filePath): + pass + else: + ctx.ui.error(_('ActionsAPI [unlink]: File %s doesn\'t exists.') % (filePath)) + +def unlinkDir(sourceDirectory): + '''delete an entire directory tree''' + if isDirectory(sourceDirectory) or isLink(sourceDirectory): + try: + shutil.rmtree(sourceDirectory) + except OSError: + error(_('ActionsAPI [unlinkDir]: Operation not permitted: %s') % (sourceDirectory)) + elif isFile(sourceDirectory): + pass + else: + error(_('ActionsAPI [unlinkDir]: Directory %s doesn\'t exists.') % (sourceDirectory)) + +def move(source, destination): + '''recursively move a "source" file or directory to "destination"''' + for filePath in glob.glob(source): + if isFile(filePath) or isLink(filePath) or isDirectory(filePath): + try: + shutil.move(filePath, destination) + except OSError: + error(_('ActionsAPI [move]: Permission denied: %s to %s') % (filePath, destination)) + else: + error(_('ActionsAPI [move]: File %s doesn\'t exists.') % (filePath)) + +# FIXME: instead of passing a sym parameter, split copy and copytree into 4 different function +def copy(source, destination, sym = True): + '''recursively copy a "source" file or directory to "destination"''' + for filePath in glob.glob(source): + if isFile(filePath) and not isLink(filePath): + try: + shutil.copy(filePath, destination) + except IOError: + error(_('ActionsAPI [copy]: Permission denied: %s to %s') % (filePath, destination)) + elif isLink(filePath) and sym: + if isDirectory(destination): + os.symlink(os.readlink(filePath), join_path(destination, os.path.basename(filePath))) + else: + if isFile(destination): + os.remove(destination) + os.symlink(os.readlink(filePath), destination) + elif isLink(filePath) and not sym: + if isDirectory(filePath): + copytree(filePath, destination) + else: + shutil.copy(filePath, destination) + elif isDirectory(filePath): + copytree(filePath, destination, sym) + else: + error(_('ActionsAPI [copy]: File %s does not exist.') % filePath) + +def copytree(source, destination, sym = True): + '''recursively copy an entire directory tree rooted at source''' + if isDirectory(source): + if os.path.exists(destination): + if isDirectory(destination): + copytree(source, join_path(destination, os.path.basename(source.strip('/')))) + return + else: + copytree(source, join_path(destination, os.path.basename(source))) + return + try: + shutil.copytree(source, destination, sym) + except OSError, e: + error(_('ActionsAPI [copytree] %s to %s: %s') % (source, destination, e)) + else: + error(_('ActionsAPI [copytree]: Directory %s doesn\'t exists.') % (source)) + +def touch(filePath): + '''changes the access time of the 'filePath', or creates it if it is not exist''' + if glob.glob(filePath): + for f in glob.glob(filePath): + os.utime(f, None) + else: + try: + f = open(filePath, 'w') + f.close() + except IOError: + error(_('ActionsAPI [touch]: Permission denied: %s') % (filePath)) + +def cd(directoryName = ''): + '''change directory''' + current = os.getcwd() + if directoryName: + os.chdir(directoryName) + else: + os.chdir(os.path.dirname(current)) + +def ls(source): + '''listdir''' + if os.path.isdir(source): + return os.listdir(source) + else: + return glob.glob(source) + +def export(key, value): + '''export environ variable''' + os.environ[key] = value + +def isLink(filePath): + '''return True if filePath refers to a symbolic link''' + return os.path.islink(filePath) + +def isFile(filePath): + '''return True if filePath is an existing regular file''' + return os.path.isfile(filePath) + +def isDirectory(filePath): + '''Return True if filePath is an existing directory''' + return os.path.isdir(filePath) + +def realPath(filePath): + '''return the canonical path of the specified filename, eliminating any symbolic links encountered in the path''' + return os.path.realpath(filePath) + +def baseName(filePath): + '''return the base name of pathname filePath''' + return os.path.basename(filePath) + +def dirName(filePath): + '''return the directory name of pathname path''' + return os.path.dirname(filePath) + +def system(command): + command = string.join(string.split(command)) + return run_logged(command) diff --git a/branches/pisi-1.1/pisi/actionsapi/variables.py b/branches/pisi-1.1/pisi/actionsapi/variables.py new file mode 100644 index 00000000..fcc17ff2 --- /dev/null +++ b/branches/pisi-1.1/pisi/actionsapi/variables.py @@ -0,0 +1,101 @@ +# -*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. + +# Standard Python Modules +import os +from copy import deepcopy + +# Pisi-Core Modules +import pisi.context as ctx + +# Set individual information, that are generally needed for ActionsAPI + +def exportFlags(): + '''General flags used in actions API.''' + + # first reset environ + os.environ = {} + os.environ = deepcopy(ctx.config.environ) + + # Build systems depend on these environment variables. That is why + # we export them instead of using as (instance) variables. + values = ctx.config.values + os.environ['HOST'] = values.build.host + os.environ['CFLAGS'] = values.build.cflags + os.environ['CXXFLAGS'] = values.build.cxxflags + os.environ['LDFLAGS'] = values.build.ldflags + os.environ['USER_LDFLAGS'] = values.build.ldflags + os.environ['JOBS'] = values.build.jobs + +class Env(object): + '''General environment variables used in actions API''' + def __init__(self): + + exportFlags() + + self.__vars = { + 'pkg_dir': 'PKG_DIR', + 'work_dir': 'WORK_DIR', + 'install_dir': 'INSTALL_DIR', + 'src_name': 'SRC_NAME', + 'src_version': 'SRC_VERSION', + 'src_release': 'SRC_RELEASE', + 'host': 'HOST', + 'cflags': 'CFLAGS', + 'cxxflags': 'CXXFLAGS', + 'ldflags': 'LDFLAGS', + 'jobs': 'JOBS' + } + + def __getattr__(self, attr): + + # Using environment variables is somewhat tricky. Each time + # you need them you need to check for their value. + if self.__vars.has_key(attr): + return os.getenv(self.__vars[attr]) + else: + return None + +class Dirs: + '''General directories used in actions API.''' + # TODO: Eventually we should consider getting these from a/the + # configuration file + doc = 'usr/share/doc' + sbin = 'usr/sbin' + man = 'usr/share/man' + info = 'usr/share/info' + data = 'usr/share' + conf = 'etc' + localstate = 'var/lib' + defaultprefix = 'usr' + + # These should be owned by object not the class. Or else Python + # will bug us with NoneType errors because of uninitialized + # context (ctx) because of the import in build.py. + def __init__(self): + self.values = ctx.config.values + self.kde = self.values.dirs.kde_dir + self.qt = self.values.dirs.qt_dir + + +# As we import this module from build.py, we can't init glb as a +# singleton here. Or else Python will bug us with NoneType errors +# because of uninitialized context (ctx) because of exportFlags(). +# +# We import this modue from build.py becase we need to reset/init glb +# for each build. # See bug #2575 +glb = None + +def initVariables(): + global glb + ctx.env = Env() + ctx.dirs = Dirs() + glb = ctx diff --git a/branches/pisi-1.1/pisi/api.py b/branches/pisi-1.1/pisi/api.py new file mode 100644 index 00000000..00d97e1b --- /dev/null +++ b/branches/pisi-1.1/pisi/api.py @@ -0,0 +1,680 @@ +# -*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. + +"""Top level PiSi interfaces. a facade to the entire PiSi system""" + +import os +import sys +import logging +import logging.handlers + +import gettext +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +import pisi +import pisi.context as ctx +import pisi.uri +import pisi.util +import pisi.dependency as dependency +import pisi.pgraph as pgraph +import pisi.packagedb +import pisi.repodb +import pisi.installdb +import pisi.sourcedb +import pisi.lockeddbshelve as shelve +import pisi.index +import pisi.config +import pisi.metadata +import pisi.file +import pisi.version +import pisi.operations +import pisi.build +import pisi.atomicoperations +import pisi.delta +import pisi.comariface +import pisi.signalhandler + +class Error(pisi.Error): + pass + +def init(database = True, write = True, + options = pisi.config.Options(), ui = None, comar = True, + stdout = None, stderr = None, + comar_sockname = None, + signal_handling = True): + """Initialize PiSi subsystem. + + You should call finalize() when your work is finished. Otherwise + you can left the database in a bad state. + + """ + + # UI comes first + + if ui is None: + # FIXME: api importing and using pisi.cli ???? + import pisi.cli + if options: + ctx.ui = pisi.cli.CLI(options.debug, options.verbose) + else: + ctx.ui = pisi.cli.CLI() + else: + ctx.ui = ui + + # FIXME: something is wrong here... see __init__.py also. Why do we import pisi.api in __init__.py + import pisi.config + ctx.config = pisi.config.Config(options) + + if os.access('%s/var/log' % ctx.config.log_dir(), os.W_OK): + handler = logging.handlers.RotatingFileHandler('%s/var/log/pisi.log' % ctx.config.log_dir()) + #handler.setLevel(logging.DEBUG) + formatter = logging.Formatter('%(asctime)-12s: %(levelname)-8s %(message)s') + handler.setFormatter(formatter) + ctx.log = logging.getLogger('pisi') + ctx.log.addHandler(handler) + ctx.loghandler = handler + ctx.log.setLevel(logging.DEBUG) + else: + ctx.log = None + + # If given define stdout and stderr. Needed by buildfarm currently + # but others can benefit from this too. + if stdout: + ctx.stdout = stdout + if stderr: + ctx.stderr = stderr + + if signal_handling: + ctx.sig = pisi.signalhandler.SignalHandler() + + # TODO: this is definitely not dynamic beyond this point! + ctx.comar = comar and not ctx.config.get_option('ignore_comar') + # This is for YALI, used in comariface.py + ctx.comar_sockname = comar_sockname + + # initialize repository databases + ctx.database = database + if database: + shelve.init_dbenv(write=write) + ctx.repodb = pisi.repodb.init() + ctx.installdb = pisi.installdb.init() + ctx.filesdb = pisi.files.FilesDB() + ctx.componentdb = pisi.component.ComponentDB() + ctx.packagedb = pisi.packagedb.init_db() + ctx.sourcedb = pisi.sourcedb.init() + else: + ctx.repodb = None + ctx.installdb = None + ctx.filesdb = None + ctx.componentdb = None + ctx.packagedb = None + ctx.sourcedb = None + ctx.ui.debug('PiSi API initialized') + ctx.initialized = True + +def finalize(): + """Close the database cleanly and do other cleanup.""" + if ctx.initialized: + ctx.disable_keyboard_interrupts() + if ctx.log: + ctx.loghandler.flush() + ctx.log.removeHandler(ctx.loghandler) + + pisi.repodb.finalize() + pisi.installdb.finalize() + if ctx.filesdb != None: + ctx.filesdb.close() + ctx.filesdb = None + if ctx.componentdb != None: + ctx.componentdb.close() + ctx.componentdb = None + if ctx.packagedb: + pisi.packagedb.finalize_db() + ctx.packagedb = None + if ctx.sourcedb: + pisi.sourcedb.finalize() + ctx.sourcedb = None + if ctx.dbenv: + ctx.dbenv.close() + ctx.dbenv_lock.close() + if ctx.build_leftover and os.path.exists(ctx.build_leftover): + os.unlink(ctx.build_leftover) + + ctx.ui.debug('PiSi API finalized') + ctx.ui.close() + ctx.initialized = False + ctx.enable_keyboard_interrupts() + +def list_installed(): + """Return a set of installed package names.""" + return set(ctx.installdb.list_installed()) + +def list_replaces(repo = None): + """Returns a dict of the replaced packages.""" + return ctx.packagedb.get_replaces(repo = repo) + +def list_available(repo = None): + """Return a set of available package names.""" + return set(ctx.packagedb.list_packages(repo = repo)) + +def list_upgradable(): + return filter(pisi.operations.is_upgradable, ctx.installdb.list_installed()) + ctx.packagedb.get_replaces().keys() + +def package_graph(A, repo = pisi.itembyrepodb.installed, ignore_installed = False): + """Construct a package relations graph. + + Graph will contain all dependencies of packages A, if ignore_installed + option is True, then only uninstalled deps will be added. + + """ + + ctx.ui.debug('A = %s' % str(A)) + + # try to construct a pisi graph of packages to + # install / reinstall + + G_f = pgraph.PGraph(ctx.packagedb, repo) # construct G_f + + # find the "install closure" graph of G_f by package + # set A using packagedb + for x in A: + G_f.add_package(x) + B = A + #state = {} + while len(B) > 0: + Bp = set() + for x in B: + pkg = ctx.packagedb.get_package(x, repo) + #print pkg + for dep in pkg.runtimeDependencies(): + if ignore_installed: + if dependency.installed_satisfies_dep(dep): + continue + if not dep.package in G_f.vertices(): + Bp.add(str(dep.package)) + G_f.add_dep(x, dep) + B = Bp + return G_f + +def generate_install_order(A): + # returns the install order of the given install package list with any extra + # dependency that is also going to be installed + G_f, order = plan_install(A, ignore_package_conflicts = True) + return order + +def generate_remove_order(A): + # returns the remove order of the given removal package list with any extra + # reverse dependency that is also going to be removed + G_f, order = plan_remove(A) + return order + +def generate_upgrade_order(A): + # returns the upgrade order of the given upgrade package list with any needed extra + # dependency + G_f, order = plan_upgrade(A) + return order + +def generate_base_upgrade(A): + # all the packages of the system.base must be installed on the system. + # method returns the currently needed system.base component install and + # upgrade needs + base = upgrade_base(A, ignore_package_conflicts = True) + return list(base) + +def generate_conflicts(A): + # returns the conflicting packages list of the to be installed packages. + # @conflicting_pkgs: conflicting and must be removed packages list to proceed + # @conflicts_inorder: list of the conflicting packages _with each other_ in the to be installed list + # @conflicting_pairs: dictionary that contains which package in the to be installed list conflicts + # with which packages + + (conflicting_pkgs, conflicts_inorder, conflicting_pairs) = pisi.conflict.calculate_conflicts(A, ctx.packagedb) + return (conflicting_pkgs, conflicts_inorder, conflicting_pairs) + +def generate_pending_order(A): + # returns pending package list in reverse topological order of dependency + G_f = pgraph.PGraph(ctx.packagedb, pisi.itembyrepodb.installed) # construct G_f + for x in A.keys(): + G_f.add_package(x) + B = A + while len(B) > 0: + Bp = set() + for x in B.keys(): + pkg = ctx.packagedb.get_package(x, pisi.itembyrepodb.installed) + for dep in pkg.runtimeDependencies(): + if dep.package in G_f.vertices(): + G_f.add_dep(x, dep) + B = Bp + if ctx.get_option('debug'): + G_f.write_graphviz(sys.stdout) + order = G_f.topological_sort() + order.reverse() + + # Bug 4211 + if ctx.componentdb.has_component('system.base'): + order = reorder_base_packages(order) + + return order + +def configure_pending(pkgs): + # start with pending packages + # configure them in reverse topological order of dependency + + packages = {} + if not pkgs: + packages = ctx.installdb.list_pending() + else: + lp = ctx.installdb.list_pending() + for pkg in pkgs: + if lp.has_key(pkg): + packages[pkg] = lp[pkg] + + order = generate_pending_order(packages) + try: + for x in order: + if ctx.installdb.is_installed(x): + pkginfo = packages[x] + pkgname = pisi.util.package_name(x, pkginfo.version, + pkginfo.release, + False, + False) + pkg_path = pisi.util.join_path(ctx.config.lib_dir(), + 'package', pkgname) + m = pisi.metadata.MetaData() + metadata_path = pisi.util.join_path(pkg_path, ctx.const.metadata_xml) + m.read(metadata_path) + # FIXME: we need a full package info here! + pkginfo.name = x + ctx.ui.notify(pisi.ui.configuring, package = pkginfo, files = None) + pisi.comariface.post_install( + pkginfo.name, + m.package.providesComar, + pisi.util.join_path(pkg_path, ctx.const.comar_dir), + pisi.util.join_path(pkg_path, ctx.const.metadata_xml), + pisi.util.join_path(pkg_path, ctx.const.files_xml), + ) + ctx.ui.notify(pisi.ui.configured, package = pkginfo, files = None) + ctx.installdb.clear_pending(x) + except ImportError: + raise Error(_("comar package is not fully installed")) + +def info(package, installed = False): + if package.endswith(ctx.const.package_suffix): + return info_file(package) + else: + metadata, files, repo = info_name(package, installed) + return metadata, files + +def info_file(package_fn): + + if not os.path.exists(package_fn): + raise Error (_('File %s not found') % package_fn) + + package = pisi.package.Package(package_fn) + package.read() + return package.metadata, package.files + +def info_name(package_name, installed=False): + """Fetch package information for the given package.""" + if installed: + package = ctx.packagedb.get_package(package_name, pisi.itembyrepodb.installed) + repo = None + else: + package, repo = ctx.packagedb.get_package_repo(package_name, pisi.itembyrepodb.repos) + + metadata = pisi.metadata.MetaData() + metadata.package = package + #FIXME: get it from sourcedb if available + metadata.source = None + #TODO: fetch the files from server if possible (wow, you maniac -- future exa) + if installed and ctx.installdb.is_installed(package.name): + try: + files = ctx.installdb.files(package.name) + except pisi.Error, e: + ctx.ui.warning(e) + files = None + else: + files = None + return metadata, files, repo + +def search_package_terms(terms, repo = pisi.itembyrepodb.all): + return search_in_packages(terms, ctx.packagedb.list_packages(repo), repo) + +def search_in_packages(terms, packages, repo = pisi.itembyrepodb.all): + + def search(package, term): + term = unicode(term).lower() + if term in unicode(package.name).lower() or \ + term in unicode(package.summary).lower() or \ + term in unicode(package.description).lower(): + return True + + found = [] + for name in packages: + pkg = ctx.packagedb.get_package(name, repo) + if terms == filter(lambda x:search(pkg, x), terms): + found.append(name) + + return found + +def check(package): + md, files = info(package, True) + corrupt = [] + for f in files.list: + if f.hash and f.type != "config" \ + and not os.path.islink('/' + f.path): + ctx.ui.info(_("Checking /%s ") % f.path, noln=True, verbose=True) + try: + if f.hash != pisi.util.sha1_file('/' + f.path): + corrupt.append(f) + ctx.ui.error(_("\nCorrupt file: %s") % f) + else: + ctx.ui.info(_("OK"), verbose=True) + except pisi.util.FileError,e: + ctx.ui.error("\n%s" % e) + return corrupt + +def index(dirs=None, output='pisi-index.xml', skip_sources=False, skip_signing=False): + """Accumulate PiSi XML files in a directory, and write an index.""" + index = pisi.index.Index() + index.distribution = None + if not dirs: + dirs = ['.'] + for repo_dir in dirs: + repo_dir = str(repo_dir) + ctx.ui.info(_('* Building index of PiSi files under %s') % repo_dir) + index.index(repo_dir, skip_sources) + + if skip_signing: + index.write(output, sha1sum=True, compress=pisi.file.File.bz2, sign=None) + else: + index.write(output, sha1sum=True, compress=pisi.file.File.bz2, sign=pisi.file.File.detached) + ctx.ui.info(_('* Index file written')) + +def add_repo(name, indexuri, at = None): + if ctx.repodb.has_repo(name): + raise Error(_('Repo %s already present.') % name) + else: + repo = pisi.repodb.Repo(pisi.uri.URI(indexuri)) + ctx.repodb.add_repo(name, repo, at = at) + ctx.ui.info(_('Repo %s added to system.') % name) + +def remove_repo(name): + if ctx.repodb.has_repo(name): + ctx.repodb.remove_repo(name) + pisi.util.clean_dir(os.path.join(ctx.config.index_dir(), name)) + ctx.ui.info(_('Repo %s removed from system.') % name) + else: + ctx.ui.error(_('Repository %s does not exist. Cannot remove.') + % name) + +def list_repos(): + return ctx.repodb.list() + +def update_repo(repo, force=False): + ctx.ui.info(_('* Updating repository: %s') % repo) + ctx.ui.notify(pisi.ui.updatingrepo, name = repo) + index = pisi.index.Index() + if ctx.repodb.has_repo(repo): + repouri = ctx.repodb.get_repo(repo).indexuri.get_uri() + try: + index.read_uri_of_repo(repouri, repo) + except pisi.file.AlreadyHaveException, e: + ctx.ui.info(_('No updates available for repository %s.') % repo) + if force: + ctx.ui.info(_('Updating database at any rate as requested')) + index.read_uri_of_repo(repouri, repo, force = force) + else: + return + + try: + index.check_signature(repouri, repo) + except pisi.file.NoSignatureFound, e: + ctx.ui.warning(e) + + ctx.txn_proc(lambda txn : index.update_db(repo, txn=txn)) + ctx.ui.info(_('* Package database updated.')) + else: + raise Error(_('No repository named %s found.') % repo) + +def delete_cache(): + pisi.util.clean_dir(ctx.config.packages_dir()) + pisi.util.clean_dir(ctx.config.archives_dir()) + pisi.util.clean_dir(ctx.config.tmp_dir()) + +def rebuild_repo(repo): + ctx.ui.info(_('* Rebuilding \'%s\' named repo... ') % repo) + + if ctx.repodb.has_repo(repo): + repouri = pisi.uri.URI(ctx.repodb.get_repo(repo).indexuri.get_uri()) + indexname = repouri.filename() + index = pisi.index.Index() + indexpath = pisi.util.join_path(ctx.config.index_dir(), repo, indexname) + tmpdir = os.path.join(ctx.config.tmp_dir(), 'index') + pisi.util.clean_dir(tmpdir) + pisi.util.check_dir(tmpdir) + try: + index.read_uri(indexpath, tmpdir, force=True) # don't look for sha1sum there + except IOError, e: + ctx.ui.warning(_("Input/Output error while reading %s: %s") % (indexpath, unicode(e))) + return + ctx.txn_proc(lambda txn : index.update_db(repo, txn=txn)) + else: + raise Error(_('No repository named %s found.') % repo) + +def rebuild_db(files=False): + + assert not ctx.database + + # Bug 2596 + # finds and cleans duplicate package directories under '/var/lib/pisi/package' + # deletes the _older_ versioned package directories. + def clean_duplicates(): + i_version = {} # installed versions + replica = [] + for pkg in os.listdir(pisi.util.join_path(pisi.api.ctx.config.lib_dir(), 'package')): + (name, ver) = pisi.util.parse_package_name(pkg) + if i_version.has_key(name): + if pisi.version.Version(ver) > pisi.version.Version(i_version[name]): + # found a greater version, older one is a replica + replica.append(name + '-' + i_version[name]) + i_version[name] = ver + else: + # found an older version which is a replica + replica.append(name + '-' + ver) + else: + i_version[name] = ver + + for pkg in replica: + pisi.util.clean_dir(pisi.util.join_path(pisi.api.ctx.config.lib_dir(), 'package', pkg)) + + def destroy(files): + #TODO: either don't delete version files here, or remove force flag... + import bsddb3.db + for db in os.listdir(ctx.config.db_dir()): + if db.endswith('.bdb'):# or db.startswith('log'): # delete only db files + if db.startswith('files') or db.startswith('filesdbversion'): + clean = files + else: + clean = True + if clean: + fn = pisi.util.join_path(ctx.config.db_dir(), db) + #NB: there is a parameter bug with python-bsddb3, fixed in pardus + ctx.dbenv.dbremove(file=fn, flags=bsddb3.db.DB_AUTO_COMMIT) + + def reload_packages(files, txn): + packages = os.listdir(pisi.util.join_path(ctx.config.lib_dir(), 'package')) + progress = ctx.ui.Progress(len(packages)) + processed = 0 + for package_fn in packages: + if not package_fn == "scripts": + ctx.ui.debug('Resurrecting %s' % package_fn) + pisi.api.resurrect_package(package_fn, files, txn) + processed += 1 + ctx.ui.display_progress(operation = "rebuilding-db", + percent = progress.update(processed), + info = _("Rebuilding package database")) + + def reload_indices(): + index_dir = ctx.config.index_dir() + if os.path.exists(index_dir): # it may have been erased, or we may be upgrading from a previous version -- exa + for repo in os.listdir(index_dir): + indexuri = pisi.util.join_path(ctx.config.lib_dir(), 'index', repo, 'uri') + indexuri = open(indexuri, 'r').readline() + pisi.api.add_repo(repo, indexuri) + pisi.api.rebuild_repo(repo) + + # check db schema versions + try: + shelve.check_dbversion('filesdbversion', pisi.__filesdbversion__, write=False) + except KeyboardInterrupt: + raise + except Exception: #FIXME: what exception could we catch here, replace with that. + files = True # exception means the files db version was wrong + shelve.init_dbenv(write=True, writeversion=True) + destroy(files) # bye bye + + # save parameters and shutdown pisi + options = ctx.config.options + ui = ctx.ui + comar = ctx.comar + finalize() + + # construct new database + init(database=True, options=options, ui=ui, comar=comar) + clean_duplicates() + txn = ctx.dbenv.txn_begin() + reload_packages(files, txn) + reload_indices() + txn.commit() + +############# FIXME: this was a quick fix. ############################## + +# api was importing other module's functions and providing them as api functions. This is wrong. +# these are quick fixes for this problem. The api functions should be in this module. + +# from pisi.operations import install, remove, upgrade, emerge +# from pisi.operations import plan_install_pkg_names as plan_install +# from pisi.operations import plan_remove, plan_upgrade, upgrade_base, calculate_conflicts, reorder_base_packages +# from pisi.build import build_until +# from pisi.atomicoperations import resurrect_package, build + +def install(*args, **kw): + return pisi.operations.install(*args, **kw) + +def remove(*args, **kw): + return pisi.operations.remove(*args, **kw) + +def upgrade(*args, **kw): + return pisi.operations.upgrade(*args, **kw) + +def emerge(*args, **kw): + return pisi.operations.emerge(*args, **kw) + +def plan_install(*args, **kw): + return pisi.operations.plan_install_pkg_names(*args, **kw) + +def plan_remove(*args, **kw): + return pisi.operations.plan_remove(*args, **kw) + +def plan_upgrade(*args, **kw): + return pisi.operations.plan_upgrade(*args, **kw) + +def upgrade_base(*args, **kw): + return pisi.operations.upgrade_base(*args, **kw) + +def calculate_conflicts(*args, **kw): + return pisi.conflict.calculate_conflicts(*args, **kw) + +def reorder_base_packages(*args, **kw): + return pisi.operations.reorder_base_packages(*args, **kw) + +def build_until(*args, **kw): + return pisi.build.build_until(*args, **kw) + +def build(*args, **kw): + return pisi.atomicoperations.build(*args, **kw) + +def resurrect_package(*args, **kw): + return pisi.atomicoperations.resurrect_package(*args, **kw) + +######################################################################## + +## Deletes the cached pisi packages to keep the package cache dir within cache limits +# @param all When set all the cached packages will be deleted +def clearCache(all=False): + + import glob + from sets import Set as set + + def getPackageLists(pkgList): + latest = {} + for f in pkgList: + try: + name, version = util.parse_package_name(f) + if latest.has_key(name): + if Version(latest[name]) < Version(version): + latest[name] = version + else: + if version: + latest[name] = version + except: + pass + + latestVersions = [] + for pkg in latest: + latestVersions.append("%s-%s" % (pkg, latest[pkg])) + + oldVersions = list(set(pkgList) - set(latestVersions)) + return oldVersions, latestVersions + + def getRemoveOrder(cacheDir, pkgList): + sizes = {} + for pkg in pkgList: + sizes[pkg] = os.stat(os.path.join(cacheDir, pkg) + ".pisi").st_size + + # sort dictionary by value from PEP-265 + from operator import itemgetter + return sorted(sizes.iteritems(), key=itemgetter(1), reverse=False) + + def removeOrderByLimit(cacheDir, order, limit): + totalSize = 0 + for pkg, size in order: + totalSize += size + if totalSize >= limit: + try: + os.remove(os.path.join(cacheDir, pkg) + ".pisi") + except exceptions.OSError: + pass + + def removeAll(cacheDir): + cached = glob.glob("%s/*.pisi" % cacheDir) + glob.glob("%s/*.part" % cacheDir) + for pkg in cached: + try: + os.remove(pkg) + except exceptions.OSError: + pass + + cacheDir = ctx.config.packages_dir() + + pkgList = map(lambda x: os.path.basename(x).split(".pisi")[0], glob.glob("%s/*.pisi" % cacheDir)) + if not all: + # Cache limits from pisi.conf + limit = int(ctx.config.values.general.package_cache_limit) * 1024 * 1024 # is this safe? + if not limit: + return + + old, latest = getPackageLists(pkgList) + order = getRemoveOrder(cacheDir, latest) + getRemoveOrder(cacheDir, old) + removeOrderByLimit(cacheDir, order, limit) + else: + removeAll(cacheDir) diff --git a/branches/pisi-1.1/pisi/archive.py b/branches/pisi-1.1/pisi/archive.py new file mode 100644 index 00000000..7395fabb --- /dev/null +++ b/branches/pisi-1.1/pisi/archive.py @@ -0,0 +1,446 @@ +# -*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. +# + +"""Archive module provides access to regular archive file types.""" + +# standard library modules +import os +import stat +import shutil +import tarfile +import zipfile +import gzip + +import gettext +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +# PiSi modules +import pisi +import pisi.util as util +import pisi.context as ctx + +class ArchiveError(pisi.Error): + pass + +class LZMAError(pisi.Error): + def __init__(self, err): + pisi.Error.__init__(self, _("An error has occured while running LZMA:\n%s") % err) + +class ArchiveBase(object): + """Base class for Archive classes.""" + def __init__(self, file_path, atype): + self.file_path = file_path + self.type = atype + + def unpack(self, target_dir, clean_dir = False): + self.target_dir = target_dir + # first we check if we need to clean-up our working env. + if os.path.exists(self.target_dir) and clean_dir: + util.clean_dir(self.target_dir) + + os.makedirs(self.target_dir) + + +class ArchiveBinary(ArchiveBase): + """ArchiveBinary handles binary archive files (usually distrubuted as + .bin files)""" + def __init__(self, file_path, arch_type = "binary"): + super(ArchiveBinary, self).__init__(file_path, arch_type) + + def unpack(self, target_dir, clean_dir = False): + super(ArchiveBinary, self).unpack(target_dir, clean_dir) + + # we can't unpack .bin files. we'll just move them to target + # directory and leave the dirty job to actions.py ;) + import shutil + target_file = os.path.join(target_dir, os.path.basename(self.file_path)) + shutil.copyfile(self.file_path, target_file) + +class ArchiveGzip(ArchiveBase): + """ArchiveGzip handles Gzip archive files""" + def __init__(self, file_path, arch_type = "gz"): + super(ArchiveGzip, self).__init__(file_path, arch_type) + + def unpack(self, target_dir, clean_dir = False): + super(ArchiveGzip, self).unpack(target_dir, clean_dir) + self.unpack_dir(target_dir) + + def unpack_dir(self, target_dir): + """Unpack Gzip archive to a given target directory(target_dir).""" + oldwd = os.getcwd() + os.chdir(target_dir) + + self.gzip = gzip.GzipFile(self.file_path, "r") + self.output = open(os.path.basename(self.file_path.rstrip(".gz")), "w") + self.output.write(self.gzip.read()) + self.output.close() + self.gzip.close() + + os.chdir(oldwd) + +class ArchiveTar(ArchiveBase): + """ArchiveTar handles tar archives depending on the compression + type. Provides access to tar, tar.gz and tar.bz2 files. + + This class provides the unpack magic for tar archives.""" + def __init__(self, file_path, arch_type = "tar", no_same_permissions = True, no_same_owner = True): + super(ArchiveTar, self).__init__(file_path, arch_type) + self.tar = None + self.no_same_permissions = no_same_permissions + self.no_same_owner = no_same_owner + + def unpack(self, target_dir, clean_dir = False): + """Unpack tar archive to a given target directory(target_dir).""" + super(ArchiveTar, self).unpack(target_dir, clean_dir) + self.unpack_dir(target_dir) + + def unpack_dir(self, target_dir): + rmode = "" + if self.type == 'tar': + rmode = 'r:' + elif self.type == 'targz': + rmode = 'r:gz' + elif self.type == 'tarbz2': + rmode = 'r:bz2' + elif self.type == 'tarlzma': + rmode = 'r:' + self.file_path = self.file_path.rstrip(ctx.const.lzma_suffix) + ret, out, err = util.run_batch("lzma d %s %s" % (self.file_path + ctx.const.lzma_suffix, + self.file_path)) + if ret != 0: + raise LZMAError(err) + else: + raise ArchiveError(_("Archive type not recognized")) + + self.tar = tarfile.open(self.file_path, rmode) + oldwd = os.getcwd() + os.chdir(target_dir) + + uid = os.getuid() + gid = os.getgid() + + install_tar_path = util.join_path(ctx.config.tmp_dir(), ctx.const.install_tar) + for tarinfo in self.tar: + # Installing packages (especially shared libraries) is a + # bit tricky. You should also change the inode if you + # change the file, cause the file is opened allready and + # accessed. Removing and creating the file will also + # change the inode and will do the trick (in fact, old + # file will be deleted only when its closed). + # + # Also, tar.extract() doesn't write on symlinks... Not any + # more :). + if self.file_path == install_tar_path: + if os.path.isfile(tarinfo.name) or os.path.islink(tarinfo.name): + try: + os.unlink(tarinfo.name) + except OSError, e: + ctx.ui.warning(e) + + self.tar.extract(tarinfo) + + # tarfile.extract does not honor umask. It must be honored explicitly. + # see --no-same-permissions option of tar(1), which is the deafult + # behaviour. + # + # Note: This is no good while installing a pisi package. Thats why + # this is optional. + if self.no_same_permissions and not os.path.islink(tarinfo.name): + os.chmod(tarinfo.name, tarinfo.mode & ~ctx.const.umask) + + if self.no_same_owner: + if not os.path.islink(tarinfo.name): + os.chown(tarinfo.name, uid, gid) + else: + os.lchown(tarinfo.name, uid, gid) + + os.chdir(oldwd) + self.close() + + def add_to_archive(self, file_name, arc_name=None): + """Add file or directory path to the tar archive""" + if not self.tar: + if self.type == 'tar': + wmode = 'w:' + elif self.type == 'targz': + wmode = 'w:gz' + elif self.type == 'tarbz2': + wmode = 'w:bz2' + elif self.type == 'tarlzma': + wmode = 'w:' + self.file_path = self.file_path.rstrip(ctx.const.lzma_suffix) + else: + raise ArchiveError(_("Archive type not recognized")) + self.tar = tarfile.open(self.file_path, wmode) + + self.tar.add(file_name, arc_name) + + def close(self): + self.tar.close() + + if self.tar.mode == 'wb' and self.type == 'tarlzma': + batch = None + if ctx.config.values.build.compressionlevel: + batch = "lzmash -%s %s" % (ctx.config.values.build.compressionlevel, self.file_path) + else: + batch = "lzmash %s" % self.file_path + + ret, out, err = util.run_batch(batch) + if ret != 0: + raise LZMAError(err) + + +class MyZipFile(zipfile.ZipFile): + def decompressToFile(self, name, outname): + import zlib + import binascii + + block_size = 1024 * 1024 * 2 + + if self.mode not in ("r", "a"): + raise RuntimeError, 'read() requires mode "r" or "a"' + if not self.fp: + raise RuntimeError, \ + "Attempt to read ZIP archive that was already closed" + zinfo = self.getinfo(name) + filepos = self.fp.tell() + self.fp.seek(zinfo.file_offset, 0) + + destfile = file(outname, 'wb') + + if zinfo.compress_type == zipfile.ZIP_STORED: + total_read = 0 + crc = None + while total_read < zinfo.compress_size: + if zinfo.compress_size - total_read < block_size: + block_size = zinfo.compress_size - total_read + buff = self.fp.read(block_size) + destfile.write(buff) + total_read += (block_size) + if crc: + crc = binascii.crc32(buff, crc) + else: + crc = binascii.crc32(buff) + destfile.close() + self.fp.seek(filepos, 0) + if crc and crc != zinfo.CRC: + raise zipfile.BadZipfile, "Bad CRC-32 for file %s" % name + return + elif zinfo.compress_type != zipfile.ZIP_DEFLATED: + raise zipfile.BadZipfile, \ + "Unsupported compression method %d for file %s" % \ + (zinfo.compress_type, name) + + if not zlib: + raise RuntimeError, \ + "De-compression requires the (missing) zlib module" + # zlib compress/decompress code by Jeremy Hylton of CNRI + dc = zlib.decompressobj(-15) + + total_read = 0 + crc = None + while total_read < zinfo.compress_size: + if zinfo.compress_size - total_read < block_size: + block_size = zinfo.compress_size - total_read + buff = self.fp.read(block_size) + dcbuff = dc.decompress(dc.unconsumed_tail + buff) + destfile.write(dcbuff) + total_read += block_size + if crc: + crc = binascii.crc32(dcbuff, crc) + else: + crc = binascii.crc32(dcbuff) + + # need to feed in unused pad byte so that zlib won't choke + ex = dc.decompress(dc.unconsumed_tail + 'Z') + dc.flush() + if ex: + if crc: + crc = binascii.crc32(ex, crc) + else: + crc = binascii.crc32(ex) + destfile.write(ex) + + if crc and crc != zinfo.CRC: + raise zipfile.BadZipfile, "Bad CRC-32 for file %s" % name + + destfile.close() + self.fp.seek(filepos, 0) + + +class ArchiveZip(ArchiveBase): + """ArchiveZip handles zip archives. + + Being a zip archive PiSi packages also use this class + extensively. This class provides unpacking and packing magic for + zip archives.""" + + symmagic = 2716663808 #long of hex val '0xA1ED0000L' + + def __init__(self, file_path, arch_type = "zip", mode = 'r'): + super(ArchiveZip, self).__init__(file_path, arch_type) + + self.zip_obj = MyZipFile(self.file_path, mode) + + def close(self): + """Close the zip archive.""" + self.zip_obj.close() + + def add_to_archive(self, file_name, arc_name=None): + """Add file or directory path to the zip file""" + # It's a pity that zipfile can't handle unicode strings. Grrr! + file_name = str(file_name) + if os.path.isdir(file_name) and not os.path.islink(file_name): + self.zip_obj.writestr(file_name + '/', '') + attr_obj = self.zip_obj.getinfo(file_name + '/') + attr_obj.external_attr = stat.S_IMODE(os.stat(file_name)[0]) << 16L + for f in os.listdir(file_name): + self.add_to_archive(os.path.join(file_name, f)) + else: + if os.path.islink(file_name): + dest = os.readlink(file_name) + attr = zipfile.ZipInfo() + attr.filename = file_name + attr.create_system = 3 + attr.external_attr = self.symmagic + self.zip_obj.writestr(attr, dest) + else: + comp_type = zipfile.ZIP_DEFLATED + if file_name.endswith(".lzma"): + comp_type = zipfile.ZIP_STORED + self.zip_obj.write(file_name, arc_name, comp_type) + + if not arc_name: + zinfo = self.zip_obj.getinfo(file_name) + else: + zinfo = self.zip_obj.getinfo(arc_name) + zinfo.create_system = 3 + + def add_basename_to_archive(self, file_name): + """Add only the basepath to the zip file. For example; if the given + file_name parameter is /usr/local/bin/somedir, this function + will create only the base directory/file somedir in the + archive.""" + cwd = os.getcwd() + path_name = os.path.dirname(file_name) + file_name = os.path.basename(file_name) + if path_name: + os.chdir(path_name) + self.add_to_archive(file_name) + os.chdir(cwd) + + def has_file(self, file_path): + """ Returns true if file_path is member of the zip archive""" + return file_path in self.zip_obj.namelist() + + def read_file(self, file_path): + return self.zip_obj.read(file_path) + + def unpack_file_cond(self, pred, target_dir, archive_root = ''): + """Unpack/Extract files according to predicate function + pred: filename -> bool + unpacks stuff into target_dir and only extracts files + from archive_root, treating it as the archive root""" + zip_obj = self.zip_obj + for info in zip_obj.infolist(): + if pred(info.filename): # check if condition holds + + # below code removes that, so we find it here + is_dir = info.filename.endswith('/') + + # calculate output file name + if archive_root == '': + outpath = info.filename + else: + # change archive_root + if util.subpath(archive_root, info.filename): + outpath = util.removepathprefix(archive_root, + info.filename) + else: + continue # don't extract if not under + + ofile = os.path.join(target_dir, outpath) + + if is_dir: # this is a directory + d = os.path.join(target_dir, outpath) + if not os.path.isdir(d): + os.makedirs(d) + perm = info.external_attr + perm &= 0xFFFF0000 + perm >>= 16 + perm |= 0x00000100 + os.chmod(d, perm) + continue + + # check that output dir is present + util.check_dir(os.path.dirname(ofile)) + + # remove output file we might be overwriting. + # (also check for islink? for broken symlinks...) + if os.path.isfile(ofile) or os.path.islink(ofile): + os.remove(ofile) + + if info.external_attr == self.symmagic: + if os.path.isdir(ofile): + shutil.rmtree(ofile) # a rare case, the file used to be a dir, now it is a symlink! + target = zip_obj.read(info.filename) + os.symlink(target, ofile) + else: + perm = info.external_attr + perm &= 0x08FF0000 + perm >>= 16 + perm |= 0x00000100 + zip_obj.decompressToFile(info.filename, ofile) + os.chmod(ofile, perm) + + def unpack_files(self, paths, target_dir): + self.unpack_file_cond(lambda f:f in paths, target_dir) + + def unpack_dir(self, path, target_dir): + self.unpack_file_cond(lambda f:util.subpath(path, f), target_dir) + + def unpack_dir_flat(self, path, target_dir): + self.unpack_file_cond(lambda f:util.subpath(path, f), target_dir, path) + + def unpack(self, target_dir, clean_dir=False): + super(ArchiveZip, self).unpack(target_dir, clean_dir) + + self.unpack_file_cond(lambda f: True, target_dir) + self.close() + return + + +class Archive: + """Archive is the main factory for ArchiveClasses, regarding the + Abstract Factory Pattern :).""" + + def __init__(self, file_path, arch_type): + """accepted archive types: + targz, tarbz2, zip, tar""" + + handlers = { + 'targz': ArchiveTar, + 'tarbz2': ArchiveTar, + 'tarlzma': ArchiveTar, + 'tar': ArchiveTar, + 'zip': ArchiveZip, + 'gzip': ArchiveGzip, + 'binary': ArchiveBinary + } + + self.archive = handlers.get(arch_type)(file_path, arch_type) + + def unpack(self, target_dir, clean_dir = False): + self.archive.unpack(target_dir, clean_dir) + + def unpack_files(self, files, target_dir): + self.archive.unpack_files(files, target_dir) diff --git a/branches/pisi-1.1/pisi/atomicoperations.py b/branches/pisi-1.1/pisi/atomicoperations.py new file mode 100644 index 00000000..99250946 --- /dev/null +++ b/branches/pisi-1.1/pisi/atomicoperations.py @@ -0,0 +1,620 @@ +# -*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. + +"""Atomic package operations such as install/remove/upgrade""" + +import gettext +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +import os +import bsddb3.db as db +import shutil + +import pisi +import pisi.context as ctx +import pisi.conflict +import pisi.util as util +import pisi.metadata +import pisi.files +import pisi.uri +import pisi.ui +import pisi.version +import pisi.delta +import pisi.packagedb + +class Error(pisi.Error): + pass + +class NotfoundError(pisi.Error): + pass + +# single package operations + +class AtomicOperation(object): + + def __init__(self, ignore_dep = None): + #self.package = package + if ignore_dep==None: + self.ignore_dep = ctx.config.get_option('ignore_dependency') + else: + self.ignore_dep = ignore_dep + + def run(self, package): + "perform an atomic package operation" + pass + + +class Install(AtomicOperation): + "Install class, provides install routines for pisi packages" + + @staticmethod + def from_name(name, ignore_dep = None): + # download package and return an installer object + # find package in repository + repo = ctx.packagedb.which_repo(name) + if repo: + ctx.ui.info(_("Package %s found in repository %s") % (name, repo)) + repo = ctx.repodb.get_repo(repo) + pkg = ctx.packagedb.get_package(name) + delta = None + + # Package is installed. This is an upgrade. Check delta. + if ctx.installdb.is_installed(pkg.name): + (version, release, build) = ctx.installdb.get_version(pkg.name) + delta = pkg.get_delta(buildFrom=build) + + # If delta exists than use the delta uri. + if delta: + pkg_uri = delta.packageURI + else: + pkg_uri = pkg.packageURI + + uri = pisi.uri.URI(pkg_uri) + if uri.is_absolute_path(): + pkg_path = str(pkg_uri) + else: + pkg_path = os.path.join(os.path.dirname(repo.indexuri.get_uri()), + str(uri.path())) + + ctx.ui.info(_("Package URI: %s") % pkg_path, verbose=True) + + return Install(pkg_path, ignore_dep) + else: + raise Error(_("Package %s not found in any active repository.") % name) + + def __init__(self, package_fname, ignore_dep = None, ignore_file_conflicts = None): + "initialize from a file name" + super(Install, self).__init__(ignore_dep) + if not ignore_file_conflicts: + ignore_file_conflicts = ctx.get_option('ignore_file_conflicts') + self.ignore_file_conflicts = ignore_file_conflicts + self.package_fname = package_fname + self.package = pisi.package.Package(package_fname) + self.package.read() + self.metadata = self.package.metadata + self.files = self.package.files + self.pkginfo = self.metadata.package + + def install(self, ask_reinstall = True): + if ctx.get_option('fetch_only'): + return + + ctx.ui.status(_('Installing %s, version %s, release %s, build %s') % + (self.pkginfo.name, self.pkginfo.version, + self.pkginfo.release, self.pkginfo.build)) + ctx.ui.notify(pisi.ui.installing, package=self.pkginfo, files=self.files) + + self.ask_reinstall = ask_reinstall + self.check_requirements() + self.check_relations() + self.check_reinstall() + self.extract_install() + + ctx.disable_keyboard_interrupts() + self.store_pisi_files() + self.postinstall() + + txn = ctx.dbenv.txn_begin() + try: + self.update_databases(txn) + txn.commit() + except db.DBError, e: + txn.abort() + raise e + + ctx.enable_keyboard_interrupts() + + ctx.ui.close() + if self.upgrade: + event = pisi.ui.upgraded + else: + event = pisi.ui.installed + ctx.ui.notify(event, package = self.pkginfo, files = self.files) + + def check_requirements(self): + """check system requirements""" + #TODO: IS THERE ENOUGH SPACE? + # what to do if / is split into /usr, /var, etc. + # check comar + if self.metadata.package.providesComar and ctx.comar: + import pisi.comariface as comariface + comariface.get_comar() + + def check_relations(self): + # check dependencies + if not ctx.config.get_option('ignore_dependency'): + if not self.pkginfo.installable(): + raise Error(_("%s package cannot be installed unless the dependencies are satisfied") % + self.pkginfo.name) + + # check if package is in database + # If it is not, put it into 3rd party packagedb + if not ctx.packagedb.has_package(self.pkginfo.name): + ctx.packagedb.add_package(self.pkginfo, pisi.itembyrepodb.thirdparty) + + # If it is explicitly specified that package conflicts with this package and also + # we passed check_conflicts tests in operations.py than this means a non-conflicting + # pkg is in "order" to be installed that has no file conflict problem with this package. + # PS: we need this because "order" generating code does not consider conflicts. + def really_conflicts(pkg): + if not self.pkginfo.conflicts: + return True + + return not pkg in map(lambda x:x.package, self.pkginfo.conflicts) + + # check file conflicts + file_conflicts = [] + for f in self.files.list: + if ctx.filesdb.has_file(f.path): + pkg, existing_file = ctx.filesdb.get_file(f.path) + dst = pisi.util.join_path(ctx.config.dest_dir(), f.path) + if pkg != self.pkginfo.name and not os.path.isdir(dst) and really_conflicts(pkg): + file_conflicts.append( (pkg, existing_file) ) + if file_conflicts: + file_conflicts_str = "" + for (pkg, existing_file) in file_conflicts: + file_conflicts_str += _("/%s from %s package\n") % (existing_file.path, pkg) + msg = _('File conflicts:\n%s') % file_conflicts_str + if self.ignore_file_conflicts: + ctx.ui.warning(msg) + else: + raise Error(msg) + + def check_reinstall(self): + "check reinstall, confirm action, and schedule reinstall" + + pkg = self.pkginfo + + self.reinstall = False + self.upgrade = False + if ctx.installdb.is_installed(pkg.name): # is this a reinstallation? + + #FIXME: consider REPOSITORY instead of DISTRIBUTION -- exa + #ipackage = ctx.packagedb.get_package(pkg.name, pisi.itembyrepodb.installed) + ipkg = ctx.installdb.get_info(pkg.name) + repomismatch = ipkg.distribution != pkg.distribution + + (iversion, irelease, ibuild) = ctx.installdb.get_version(pkg.name) + + # determine if same version + self.same_ver = False + ignore_build = ctx.config.options and ctx.config.options.ignore_build_no + if repomismatch or (not ibuild) or (not pkg.build) or ignore_build: + # we don't look at builds to compare two package versions + if pisi.version.Version(pkg.release) == pisi.version.Version(irelease): + self.same_ver = True + else: + if pkg.build == ibuild: + self.same_ver = True + + if self.same_ver: + if self.ask_reinstall: + if not ctx.ui.confirm(_('Re-install same version package?')): + raise Error(_('Package re-install declined')) + else: + upgrade = False + # is this an upgrade? + # determine and report the kind of upgrade: version, release, build + if pisi.version.Version(pkg.version) > pisi.version.Version(iversion): + ctx.ui.info(_('Upgrading to new upstream version')) + upgrade = True + elif pisi.version.Version(pkg.release) > pisi.version.Version(irelease): + ctx.ui.info(_('Upgrading to new distribution release')) + upgrade = True + elif ((not ignore_build) and ibuild and pkg.build + and pkg.build > ibuild): + ctx.ui.info(_('Upgrading to new distribution build')) + upgrade = True + self.upgrade = upgrade + + # is this a downgrade? confirm this action. + if self.ask_reinstall and (not upgrade): + if pisi.version.Version(pkg.version) < pisi.version.Version(iversion): + #x = _('Downgrade to old upstream version?') + x = None + elif pisi.version.Version(pkg.release) < pisi.version.Version(irelease): + x = _('Downgrade to old distribution release?') + else: + x = _('Downgrade to old distribution build?') + if x and not ctx.ui.confirm(x): + raise Error(_('Package downgrade declined')) + + # schedule for reinstall + self.old_files = ctx.installdb.files(pkg.name) + self.old_path = ctx.installdb.pkg_dir(pkg.name, iversion, irelease) + self.reinstall = True + self.remove_old = Remove(pkg.name) + self.remove_old.run_preremove() + + def postinstall(self): + self.config_later = False + if ctx.comar: + import pisi.comariface + try: + ctx.ui.notify(pisi.ui.configuring, package = self.pkginfo, files = self.files) + pisi.comariface.post_install( + self.pkginfo.name, + self.metadata.package.providesComar, + self.package.comar_dir(), + os.path.join(self.package.pkg_dir(), ctx.const.metadata_xml), + os.path.join(self.package.pkg_dir(), ctx.const.files_xml), + ) + ctx.ui.notify(pisi.ui.configured, package = self.pkginfo, files = self.files) + except pisi.comariface.Error: + ctx.ui.warning(_('%s configuration failed.') % self.pkginfo.name) + self.config_later = True + else: + self.config_later = True + + def extract_install(self): + "unzip package in place" + + ctx.ui.notify(pisi.ui.extracting, package = self.pkginfo, files = self.files) + + config_changed = [] + def check_config_changed(config): + changed = False + fpath = pisi.util.join_path(ctx.config.dest_dir(), config.path) + if os.path.exists(fpath) and not os.path.isdir(fpath): + if os.path.islink(fpath): + f = os.readlink(fpath) + if os.path.exists(f) and pisi.util.sha1_data(f) != config.hash: + changed = True + else: + if pisi.util.sha1_file(fpath) != config.hash: + changed = True + + if changed: + config_changed.append(fpath) + if os.path.exists(fpath + '.old'): + os.unlink(fpath + '.old') + os.rename(fpath, fpath + '.old') + + # old config files are kept as they are. New config files from the installed + # packages are saved with ".newconfig" string appended to their names. + def rename_configs(): + for path in config_changed: + newconfig = path + '.newconfig' + oldconfig = path + '.old' + if os.path.exists(newconfig): + os.unlink(newconfig) + + # In the case of delta packages: the old package and the new package + # may contain same config typed files with same hashes, so the delta + # package will not have that config file. In order to protect user + # changed config files, they are renamed with ".old" prefix in case + # of the hashes of these files on the filesystem and the new config + # file that is coming from the new package. But in delta package case + # with the given scenario there wont be any, so we can pass this one. + # If the config files were not be the same between these packages the + # delta package would have it and extract it and the path would point + # to that new config file. If they are same and the user had changed + # that file and using the changed config file, there is no problem + # here. + if os.path.exists(path): + os.rename(path, newconfig) + + os.rename(oldconfig, path) + + # Delta package does not contain the files that have the same hash as in + # the old package's. Because it means the file has not changed. But some + # of these files may be relocated to some other directory in the new package. + # We handle these cases here. + def relocate_files(): + for old_file, new_file in pisi.delta.find_relocations(self.old_files, self.files): + old_path, new_path = ("/" + old_file.path, "/" + new_file.path) + + destdir = os.path.dirname(new_path) + if not os.path.exists(destdir): + os.makedirs(destdir) + + if os.path.islink(old_path): + if not os.path.lexists(new_path): + os.symlink(os.readlink(old_path), new_path) + else: + shutil.copy(old_path, new_path) + + # remove left over files from the old package. + def clean_leftovers(): + new = set(str(f.path) for f in self.files.list) + old = set(str(f.path) for f in self.old_files.list) + leftover = old - new + old_fileinfo = {} + for fileinfo in self.old_files.list: + old_fileinfo[str(fileinfo.path)] = fileinfo + for path in leftover: + Remove.remove_file(old_fileinfo[path], self.pkginfo.name) + + if self.reinstall: + # get 'config' typed file objects + new = filter(lambda x: x.type == 'config', self.files.list) + old = filter(lambda x: x.type == 'config', self.old_files.list) + + # get config path lists + newconfig = set(str(x.path) for x in new) + oldconfig = set(str(x.path) for x in old) + + config_overlaps = newconfig & oldconfig + if config_overlaps: + files = filter(lambda x: x.path in config_overlaps, old) + for f in files: + check_config_changed(f) + else: + for f in self.files.list: + if f.type == 'config': + # there may be left over config files + check_config_changed(f) + + if self.package_fname.endswith(ctx.const.delta_package_suffix): + relocate_files() + + self.package.extract_install(ctx.config.dest_dir()) + + if config_changed: + rename_configs() + + if self.reinstall: + clean_leftovers() + + + def store_pisi_files(self): + """put files.xml, metadata.xml, actions.py and COMAR scripts + somewhere in the file system. We'll need these in future...""" + + if self.reinstall: + util.clean_dir(self.old_path) + + ctx.ui.info(_('Storing %s, ') % ctx.const.files_xml, verbose=True) + self.package.extract_file(ctx.const.files_xml, self.package.pkg_dir()) + + ctx.ui.info(_('Storing %s.') % ctx.const.metadata_xml, verbose=True) + self.package.extract_file(ctx.const.metadata_xml, self.package.pkg_dir()) + + for pcomar in self.metadata.package.providesComar: + fpath = os.path.join(ctx.const.comar_dir, pcomar.script) + # comar prefix is added to the pkg_dir while extracting comar + # script file. so we'll use pkg_dir as destination. + ctx.ui.info(_('Storing %s') % fpath, verbose=True) + self.package.extract_file(fpath, self.package.pkg_dir()) + + def update_databases(self, txn): + "update databases" + if self.reinstall: + self.remove_old.remove_db(txn) + + # installdb + ctx.installdb.install(self.metadata.package.name, + self.metadata.package.version, + self.metadata.package.release, + self.metadata.package.build, + self.metadata.package.distribution, + config_later = self.config_later, + txn = txn) + + # filesdb + ctx.filesdb.add_files(self.metadata.package.name, self.files, txn=txn) + + # installed packages + ctx.packagedb.add_package(self.pkginfo, pisi.itembyrepodb.installed, txn=txn) + + +def install_single(pkg, upgrade = False): + """install a single package from URI or ID""" + url = pisi.uri.URI(pkg) + # Check if we are dealing with a remote file or a real path of + # package filename. Otherwise we'll try installing a package from + # the package repository. + if url.is_remote_file() or os.path.exists(url.uri): + install_single_file(pkg, upgrade) + else: + install_single_name(pkg, upgrade) + +# FIXME: Here and elsewhere pkg_location must be a URI +def install_single_file(pkg_location, upgrade = False): + """install a package file""" + Install(pkg_location).install(not upgrade) + +def install_single_name(name, upgrade = False): + """install a single package from ID""" + install = Install.from_name(name) + install.install(not upgrade) + +class Remove(AtomicOperation): + + def __init__(self, package_name, ignore_dep = None): + super(Remove, self).__init__(ignore_dep) + self.package_name = package_name + self.package = ctx.packagedb.get_package(self.package_name, pisi.itembyrepodb.installed) + try: + self.files = ctx.installdb.files(self.package_name) + except pisi.Error, e: + # for some reason file was deleted, we still allow removes! + ctx.ui.error(unicode(e)) + ctx.ui.warning(_('File list could not be read for package %s, continuing removal.') % package_name) + self.files = pisi.files.Files() + + def run(self): + """Remove a single package""" + + ctx.ui.status(_('Removing package %s') % self.package_name) + ctx.ui.notify(pisi.ui.removing, package = self.package, files = self.files) + if not ctx.installdb.is_installed(self.package_name): + raise Exception(_('Trying to remove nonexistent package ') + + self.package_name) + + self.check_dependencies() + + self.run_preremove() + for fileinfo in self.files.list: + self.remove_file(fileinfo, self.package_name) + + txn = ctx.dbenv.txn_begin() + try: + self.remove_db(txn) + txn.commit() + except db.DBError, e: + txn.abort() + raise e + + self.remove_pisi_files() + ctx.ui.close() + ctx.ui.notify(pisi.ui.removed, package = self.package, files = self.files) + + def check_dependencies(self): + #FIXME: why is this not implemented? -- exa + #we only have to check the dependencies to ensure the + #system will be consistent after this removal + pass + # is there any package who depends on this package? + + @staticmethod + def remove_file(fileinfo, package_name): + fpath = pisi.util.join_path(ctx.config.dest_dir(), fileinfo.path) + + # we should check if the file belongs to another + # package (this can legitimately occur while upgrading + # two packages such that a file has moved from one package to + # another as in #2911) + if ctx.filesdb.has_file(fileinfo.path): + pkg, existing_file = ctx.filesdb.get_file(fileinfo.path) + if pkg != package_name: + ctx.ui.warning(_('Not removing conflicted file : %s') % fpath) + return + + if fileinfo.permanent: + # do not remove precious files :) + pass + elif fileinfo.type == ctx.const.conf: + # config files are precious, leave them as they are + # unless they are the same as provided by package. + try: + if pisi.util.sha1_file(fpath) == fileinfo.hash: + os.unlink(fpath) + except pisi.util.FileError: + pass + else: + if os.path.isfile(fpath) or os.path.islink(fpath): + os.unlink(fpath) + elif os.path.isdir(fpath) and not os.listdir(fpath): + os.rmdir(fpath) + else: + ctx.ui.warning(_('Installed file %s is not exists on system [Probably you manually deleted]') % fpath) + return + + # remove emptied directories + dpath = os.path.dirname(fpath) + while dpath != '/' and not os.listdir(dpath): + os.rmdir(dpath) + dpath = os.path.dirname(dpath) + + def run_preremove(self): + if ctx.comar: + import pisi.comariface + pisi.comariface.pre_remove( + self.package_name, + os.path.join(self.package.pkg_dir(), ctx.const.metadata_xml), + os.path.join(self.package.pkg_dir(), ctx.const.files_xml), + ) + + def remove_pisi_files(self): + util.clean_dir(self.package.pkg_dir()) + + def remove_db(self, txn): + ctx.installdb.remove(self.package_name, txn) + ctx.filesdb.remove_files(self.files, txn) + # FIXME: something goes wrong here, if we use ctx operations ends up with segmentation fault! + pisi.packagedb.remove_tracking_package(self.package_name, txn) + + +def remove_single(package_name): + Remove(package_name).run() + +def build(package): + # wrapper for build op + import pisi.build + return pisi.build.build(package) + +def virtual_install(metadata, files, txn): + """Recreate the package info for rebuilddb command""" + # installdb + ctx.installdb.install(metadata.package.name, + metadata.package.version, + metadata.package.release, + metadata.package.build, + metadata.package.distribution, + rebuild=True, + txn=txn) + + # filesdb + if files: + ctx.filesdb.add_files(metadata.package.name, files, txn=txn) + + # installed packages + ctx.packagedb.add_package(metadata.package, pisi.itembyrepodb.installed, txn=txn) + +def resurrect_package(package_fn, write_files, txn = None): + """Resurrect the package from xml files""" + + metadata_xml = util.join_path(ctx.config.lib_dir(), 'package', + package_fn, ctx.const.metadata_xml) + if not os.path.exists(metadata_xml): + raise Error, _("Metadata XML '%s' cannot be found") % metadata_xml + + metadata = pisi.metadata.MetaData() + metadata.read(metadata_xml) + + errs = metadata.errors() + if errs: + util.print_errors(errs) + raise Error, _("MetaData format wrong (%s)") % package_fn + + ctx.ui.info(_('* Adding \'%s\' to db... ') % (metadata.package.name), noln=True) + + if write_files: + files_xml = util.join_path(ctx.config.lib_dir(), 'package', + package_fn, ctx.const.files_xml) + if not os.path.exists(files_xml): + raise Error, _("Files XML '%s' cannot be found") % files_xml + + files = pisi.files.Files() + files.read(files_xml) + if files.errors(): + raise Error, _("Invalid %s") % ctx.const.files_xml + else: + files = None + + #import pisi.atomicoperations + def f(t): + pisi.atomicoperations.virtual_install(metadata, files, t) + ctx.txn_proc(f, txn) + + ctx.ui.info(_('OK.')) diff --git a/branches/pisi-1.1/pisi/build.py b/branches/pisi-1.1/pisi/build.py new file mode 100644 index 00000000..06e1c91a --- /dev/null +++ b/branches/pisi-1.1/pisi/build.py @@ -0,0 +1,1040 @@ +# -*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. + +"""package building code""" + +# python standard library +import os +import glob +import copy +import stat +import pwd +import grp + +import gettext +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +import pisi +import pisi.specfile +import pisi.util +import pisi.file +import pisi.context as ctx +import pisi.dependency as dependency +import pisi.operations as operations +import pisi.sourcearchive +import pisi.files +import pisi.fetcher +import pisi.uri +import pisi.metadata +import pisi.package +import pisi.component as component +import pisi.archive as archive +import pisi.actionsapi.variables + + +class Error(pisi.Error): + pass + + +# Helper Functions +def get_file_type(path, pinfo_list, install_dir): + """Return the file type of a path according to the given PathInfo + list""" + + Match = lambda x: [match for match in glob.glob(install_dir + x) if pisi.util.join_path(install_dir, path).find(match) > -1] + + def Sort(x): + x.sort(reverse=True) + return x + + best_matched_path = Sort([pinfo.path for pinfo in pinfo_list if Match(pinfo.path)])[0] + info = [pinfo for pinfo in pinfo_list if best_matched_path == pinfo.path][0] + return info.fileType, info.permanent + +def check_path_collision(package, pkgList): + """This function will check for collision of paths in a package with + the paths of packages in pkgList. The return value will be the + list containing the paths that collide.""" + collisions = [] + for pinfo in package.files: + for pkg in pkgList: + if pkg is package: + continue + for path in pkg.files: + # if pinfo.path is a subpath of path.path like + # the example below. path.path is marked as a + # collide. Exp: + # pinfo.path: /usr/share + # path.path: /usr/share/doc + if (path.path.endswith(ctx.const.ar_file_suffix) and ctx.get_option('create_static')) or \ + (path.path.endswith(ctx.const.debug_file_suffix) and ctx.config.values.build.generatedebug): + # don't throw collision error for these files. + # we'll handle this in gen_files_xml.. + continue + if pisi.util.subpath(pinfo.path, path.path): + collisions.append(path.path) + ctx.ui.debug(_('Path %s belongs in multiple packages') % + path.path) + return collisions + + +class Builder: + """Provides the package build and creation routines""" + #FIXME: this class and every other class must use URLs as paths! + + @staticmethod + def from_name(name): + # download package and return an installer object + # find package in repository + sf, reponame = ctx.sourcedb.get_spec_repo(name) + src = sf.source + if src: + + src_uri = pisi.uri.URI(src.sourceURI) + if src_uri.is_absolute_path(): + src_path = str(src_uri) + else: + repo = ctx.repodb.get_repo(reponame) + #FIXME: don't use dirname to work on URLs + src_path = os.path.join(os.path.dirname(repo.indexuri.get_uri()), + str(src_uri.path())) + + ctx.ui.debug(_("Source URI: %s") % src_path) + + return Builder(src_path) + else: + raise Error(_("Source %s not found in any active repository.") % name) + + def __init__(self, specuri): + + # process args + if not isinstance(specuri, pisi.uri.URI): + specuri = pisi.uri.URI(specuri) + + # read spec file, we'll need it :) + self.set_spec_file(specuri) + + if specuri.is_remote_file(): + self.specdir = self.fetch_files() + else: + self.specdir = os.path.dirname(self.specuri.get_uri()) + + self.read_translations(self.specdir) + + self.sourceArchive = pisi.sourcearchive.SourceArchive(self.spec, self.pkg_work_dir()) + + self.set_environment_vars() + + self.actionLocals = None + self.actionGlobals = None + self.srcDir = None + + def set_spec_file(self, specuri): + if not specuri.is_remote_file(): + specuri = pisi.uri.URI(os.path.realpath(specuri.get_uri())) # FIXME: doesn't work for file:// + self.specuri = specuri + spec = pisi.specfile.SpecFile() + spec.read(self.specuri, ctx.config.tmp_dir()) + self.spec = spec + + def read_translations(self, specdir): + self.spec.read_translations(pisi.util.join_path(specdir, ctx.const.translations_file)) + + # directory accessor functions + + # pkg_x_dir: per package directory for storing info type x + + def pkg_dir(self): + "package build directory" + packageDir = self.spec.source.name + '-' + \ + self.spec.getSourceVersion() + '-' + self.spec.getSourceRelease() + return pisi.util.join_path(ctx.config.dest_dir(), ctx.config.values.dirs.tmp_dir, + packageDir) + + def pkg_work_dir(self): + return self.pkg_dir() + ctx.const.work_dir_suffix + + def pkg_debug_dir(self): + return self.pkg_dir() + ctx.const.debug_dir_suffix + + def pkg_install_dir(self): + return self.pkg_dir() + ctx.const.install_dir_suffix + + def set_state(self, state): + stateFile = pisi.util.join_path(self.pkg_work_dir(), "pisiBuildState") + open(stateFile, "w").write(state) + + def get_state(self): + stateFile = pisi.util.join_path(self.pkg_work_dir(), "pisiBuildState") + if not os.path.exists(stateFile): # no state + return None + return open(stateFile, "r").read() + + def build(self): + """Build the package in one shot.""" + + ctx.ui.status(_("Building PiSi source package: %s") % self.spec.source.name) + + self.compile_action_script() + self.compile_comar_script() + + # check if all patch files exists, if there are missing no need to unpack! + self.patch_exists() + + self.check_build_dependencies() + self.fetch_component() + self.fetch_source_archive() + self.unpack_source_archive() + + self.run_setup_action() + self.run_build_action() + if ctx.get_option('debug') and not ctx.get_option('ignore_check'): + self.run_check_action() + self.run_install_action() + + # after all, we are ready to build/prepare the packages + return self.build_packages() + + def set_environment_vars(self): + """Sets the environment variables for actions API to use""" + + # Each time a builder is created we must reset + # environment. See bug #2575 + pisi.actionsapi.variables.initVariables() + + env = { + "PKG_DIR": self.pkg_dir(), + "WORK_DIR": self.pkg_work_dir(), + "INSTALL_DIR": self.pkg_install_dir(), + "SRC_NAME": self.spec.source.name, + "SRC_VERSION": self.spec.getSourceVersion(), + "SRC_RELEASE": self.spec.getSourceRelease() + } + os.environ.update(env) + + # First check icecream, if not found use ccache, no need to use both + # together (according to kde-wiki it cause performance loss) + if ctx.config.values.build.buildhelper == "icecream": + if os.path.exists("/opt/icecream/bin/gcc"): + # Add icecream directory for support distributed compiling :) + os.environ["PATH"] = "/opt/icecream/bin/:%s" % os.environ["PATH"] + ctx.ui.info(_("IceCream detected. Make sure your daemon is up and running...")) + elif ctx.config.values.build.buildhelper == "ccache": + if os.path.exists("/usr/lib/ccache/bin/gcc"): + # Add ccache directory for support Compiler Cache :) + os.environ["PATH"] = "/usr/lib/ccache/bin/:%s" % os.environ["PATH"] + ctx.ui.info(_("CCache detected...")) + + def fetch_files(self): + self.specdiruri = os.path.dirname(self.specuri.get_uri()) + pkgname = os.path.basename(self.specdiruri) + self.destdir = pisi.util.join_path(ctx.config.tmp_dir(), pkgname) + #self.location = os.path.dirname(self.url.uri) + + self.fetch_actionsfile() + self.fetch_translationsfile() + self.fetch_patches() + self.fetch_comarfiles() + self.fetch_additionalFiles() + + return self.destdir + + def fetch_actionsfile(self): + actionsuri = pisi.util.join_path(self.specdiruri, ctx.const.actions_file) + self.download(actionsuri, self.destdir) + + def fetch_translationsfile(self): + translationsuri = pisi.util.join_path(self.specdiruri, ctx.const.translations_file) + try: + self.download(translationsuri, self.destdir) + except pisi.fetcher.FetchError: + # translations.xml is not mandatory for PiSi + pass + + def fetch_patches(self): + spec = self.spec + for patch in spec.source.patches: + file_name = os.path.basename(patch.filename) + dir_name = os.path.dirname(patch.filename) + patchuri = pisi.util.join_path(self.specdiruri, + ctx.const.files_dir, dir_name, file_name) + self.download(patchuri, pisi.util.join_path(self.destdir, ctx.const.files_dir, dir_name)) + + def fetch_comarfiles(self): + spec = self.spec + for package in spec.packages: + for pcomar in package.providesComar: + comaruri = pisi.util.join_path(self.specdiruri, + ctx.const.comar_dir, pcomar.script) + self.download(comaruri, pisi.util.join_path(self.destdir, ctx.const.comar_dir)) + + def fetch_additionalFiles(self): + spec = self.spec + for pkg in spec.packages: + for afile in pkg.additionalFiles: + file_name = os.path.basename(afile.filename) + dir_name = os.path.dirname(afile.filename) + afileuri = pisi.util.join_path(self.specdiruri, + ctx.const.files_dir, dir_name, file_name) + self.download(afileuri, pisi.util.join_path(self.destdir, ctx.const.files_dir, dir_name)) + + def download(self, uri, transferdir): + # fix auth info and download + uri = pisi.file.File.make_uri(uri) + pisi.file.File.download(uri, transferdir) + + def fetch_component(self): + if not self.spec.source.partOf: + ctx.ui.warning(_('PartOf tag not defined, looking for component')) + diruri = pisi.util.parenturi(self.specuri.get_uri()) + parentdir = pisi.util.parenturi(diruri) + url = pisi.util.join_path(parentdir, 'component.xml') + progress = ctx.ui.Progress + if pisi.uri.URI(url).is_remote_file(): + pisi.fetcher.fetch_url(url, self.pkg_work_dir(), progress) + path = pisi.util.join_path(self.pkg_work_dir(), 'component.xml') + else: + if not os.path.exists(url): + raise Exception(_('Cannot find component.xml in upper directory')) + path = url + comp = component.Component() + comp.read(path) + ctx.ui.info(_('Source is part of %s component') % comp.name) + self.spec.source.partOf = comp.name + + def fetch_source_archive(self): + ctx.ui.info(_("Fetching source from: %s") % self.spec.source.archive.uri) + self.sourceArchive.fetch() + ctx.ui.info(_("Source archive is stored: %s/%s") + %(ctx.config.archives_dir(), self.spec.source.archive.name)) + + def unpack_source_archive(self): + ctx.ui.info(_("Unpacking archive...")) + self.sourceArchive.unpack() + # apply the patches and prepare a source directory for build. + if self.apply_patches(): + ctx.ui.info(_(" unpacked (%s)") % self.pkg_work_dir()) + self.set_state("unpack") + + def run_setup_action(self): + # Run configure, build and install phase + ctx.ui.action(_("Setting up source...")) + if self.run_action_function(ctx.const.setup_func): + self.set_state("setupaction") + + def run_build_action(self): + ctx.ui.action(_("Building source...")) + if self.run_action_function(ctx.const.build_func): + self.set_state("buildaction") + + def run_check_action(self): + ctx.ui.action(_("Testing package...")) + self.run_action_function(ctx.const.check_func) + + def run_install_action(self): + ctx.ui.action(_("Installing...")) + + # Before install make sure install_dir is clean + if os.path.exists(self.pkg_install_dir()): + pisi.util.clean_dir(self.pkg_install_dir()) + + # install function is mandatory! + if self.run_action_function(ctx.const.install_func, True): + self.set_state("installaction") + + def get_abandoned_files(self): + # return the files those are not collected from the install dir + + install_dir = self.pkg_dir() + ctx.const.install_dir_suffix + abandoned_files = [] + all_paths_in_packages = [] + + + for package in self.spec.packages: + for path in package.files: + map(lambda p: all_paths_in_packages.append(p), [p for p in glob.glob(install_dir + path.path)]) + + for root, dirs, files in os.walk(install_dir): + for file_ in files: + already_in_package = False + fpath = pisi.util.join_path(root, file_) + for path in all_paths_in_packages: + if not fpath.find(path): + already_in_package = True + if not already_in_package: + abandoned_files.append(fpath) + + return abandoned_files + + def compile_action_script(self): + """Compiles given actions.py to check syntax error in it and sets the actionLocals and actionGlobals""" + fname = pisi.util.join_path(self.specdir, ctx.const.actions_file) + try: + localSymbols = globalSymbols = {} + buf = open(fname).read() + exec compile(buf, "error", "exec") in localSymbols, globalSymbols + except IOError, e: + raise Error(_("Unable to read Actions Script (%s): %s") %(fname,e)) + except SyntaxError, e: + raise Error(_("SyntaxError in Actions Script (%s): %s") %(fname,e)) + + self.actionLocals = localSymbols + self.actionGlobals = globalSymbols + self.srcDir = self.pkg_src_dir() + + def compile_comar_script(self): + """Compiles comar scripts to check syntax errors""" + for package in self.spec.packages: + for pcomar in package.providesComar: + fname = pisi.util.join_path(self.specdir, ctx.const.comar_dir, + pcomar.script) + + try: + buf = open(fname).read() + compile(buf, "error", "exec") + except IOError, e: + raise Error(_("Unable to read COMAR script (%s): %s") %(fname,e)) + except SyntaxError, e: + raise Error(_("SyntaxError in COMAR file (%s): %s") %(fname,e)) + + def pkg_src_dir(self): + """Returns the real path of WorkDir for an unpacked archive.""" + try: + workdir = self.actionGlobals['WorkDir'] + except KeyError: + workdir = self.spec.source.name + "-" + self.spec.getSourceVersion() + + return pisi.util.join_path(self.pkg_work_dir(), workdir) + + def log_sandbox_violation(self, operation, path, canonical_path): + ctx.ui.error(_("Sandbox violation: %s (%s -> %s)") % (operation, path, canonical_path)) + + def run_action_function(self, func, mandatory=False): + """Calls the corresponding function in actions.py. + + If mandatory parameter is True, and function is not present in + actionLocals pisi.build.Error will be raised.""" + # we'll need our working directory after actionscript + # finished its work in the archive source directory. + curDir = os.getcwd() + os.chdir(self.srcDir) + + if func in self.actionLocals: + if ctx.get_option('ignore_sandbox') or not ctx.config.values.build.enablesandbox: + self.actionLocals[func]() + else: + import catbox + # stupid autoconf family needs /usr/lib/conftest* and /usr/lib/cf* for some conftest, + # http://sources.gentoo.org/viewcvs.py/portage/trunk/sandbox/files/sandbox/sandbox.c also permits these + valid_dirs = [self.pkg_dir(), "/tmp/", "/var/tmp/", "/dev/tty", "/dev/pts/", "/dev/pty", "/dev/null", "/dev/zero", "/dev/ptmx", "/proc/", "/usr/lib/conftest", "/usr/lib/cf"] + if ctx.config.values.build.buildhelper == "ccache": + valid_dirs.append("%s/.ccache" % os.environ["HOME"]) + # every qt/KDE application check these + valid_dirs.append("%s/.qt/.qt_plugins_3.3rc.lock" % os.environ["HOME"]) + valid_dirs.append("%s/.qt/qt_plugins_3.3rc.tmp" % os.environ["HOME"]) + valid_dirs.append("%s/.qt/.qtrc.lock" % os.environ["HOME"]) + valid_dirs.append("%s/.qt/.qt_designerrc.lock" % os.environ["HOME"]) + valid_dirs.append("/usr/qt/3/etc/settings/.qt_plugins_3.3rc.lock") + valid_dirs.append("/usr/qt/3/etc/settings/qt_plugins_3.3rc.tmp") + valid_dirs.append("/usr/qt/3/etc/settings/qt_plugins_3.3rc") + ret = catbox.run(self.actionLocals[func], valid_dirs, logger=self.log_sandbox_violation) + if ret.code == 1: + raise RuntimeError + if ret.violations != []: + ctx.ui.error(_("Sandbox violations!")) + else: + if mandatory: + raise Error(_("unable to call function from actions: %s") % func) + + os.chdir(curDir) + return True + + def check_build_dependencies(self): + """check and try to install build dependencies, otherwise fail.""" + + build_deps = self.spec.source.buildDependencies + + if not ctx.get_option('ignore_safety'): + if ctx.componentdb.has_component('system.devel'): + build_deps_names = set([x.package for x in build_deps]) + devel_deps_names = set(ctx.componentdb.get_component('system.devel').packages) + extra_names = devel_deps_names - build_deps_names + extra_names = filter(lambda x: not ctx.installdb.is_installed(x), extra_names) + if extra_names: + ctx.ui.warning(_('Safety switch: following extra packages in system.devel will be installed: ') + + pisi.util.strlist(extra_names)) + extra_deps = [dependency.Dependency(package = x) for x in extra_names] + build_deps.extend(extra_deps) + else: + ctx.ui.warning(_('Safety switch: system.devel is already installed')) + else: + ctx.ui.warning(_('Safety switch: the component system.devel cannot be found')) + + # find out the build dependencies that are not satisfied... + dep_unsatis = [] + for dep in build_deps: + if not dependency.installed_satisfies_dep(dep): + dep_unsatis.append(dep) + + if dep_unsatis: + ctx.ui.info(_("Unsatisfied Build Dependencies:") + ' ' + + pisi.util.strlist([str(x) for x in dep_unsatis]) ) + + def fail(): + raise Error(_('Cannot build package due to unsatisfied build dependencies')) + + if ctx.config.get_option('no_install'): + fail() + + if not ctx.config.get_option('ignore_dependency'): + for dep in dep_unsatis: + if not dependency.repo_satisfies_dep(dep): + raise Error(_('Build dependency %s cannot be satisfied') % str(dep)) + if ctx.ui.confirm( + _('Do you want to install the unsatisfied build dependencies')): + ctx.ui.info(_('Installing build dependencies.')) + operations.install([dep.package for dep in dep_unsatis]) + else: + fail() + else: + ctx.ui.warning(_('Ignoring build dependencies.')) + + def patch_exists(self): + """check existence of patch files declared in PSPEC""" + + files_dir = os.path.abspath(pisi.util.join_path(self.specdir, + ctx.const.files_dir)) + for patch in self.spec.source.patches: + patchFile = pisi.util.join_path(files_dir, patch.filename) + if not os.access(patchFile, os.F_OK): + raise Error(_("Patch file is missing: %s\n") % patch.filename) + + def apply_patches(self): + files_dir = os.path.abspath(pisi.util.join_path(self.specdir, + ctx.const.files_dir)) + + for patch in self.spec.source.patches: + patchFile = pisi.util.join_path(files_dir, patch.filename) + if patch.compressionType: + patchFile = pisi.util.uncompress(patchFile, + compressType=patch.compressionType, + targetDir=ctx.config.tmp_dir()) + + ctx.ui.action(_("* Applying patch: %s") % patch.filename) + pisi.util.do_patch(self.srcDir, patchFile, level=patch.level) + return True + return True + + def generate_static_package_object(self): + ar_files = [] + for root, dirs, files in os.walk(self.pkg_install_dir()): + for f in files: + if f.endswith(ctx.const.ar_file_suffix) and pisi.util.is_ar_file(pisi.util.join_path(root, f)): + ar_files.append(pisi.util.join_path(root, f)) + + if not len(ar_files): + return None + + static_package_obj = pisi.specfile.Package() + static_package_obj.name = self.spec.source.name + ctx.const.static_name_suffix + # FIXME: find a better way to deal with the summary and description constants. + static_package_obj.summary['en'] = u'Ar files for %s' % (self.spec.source.name) + static_package_obj.description['en'] = u'Ar files for %s' % (self.spec.source.name) + static_package_obj.partOf = self.spec.source.partOf + for f in ar_files: + static_package_obj.files.append(pisi.specfile.Path(path = f[len(self.pkg_install_dir()):], fileType = "library")) + + # append all generated packages to dependencies + for p in self.spec.packages: + static_package_obj.packageDependencies.append( + pisi.dependency.Dependency(package = p.name)) + + return static_package_obj + + def generate_debug_package_object(self): + debug_files = [] + for root, dirs, files in os.walk(self.pkg_debug_dir()): + for f in files: + if f.endswith(ctx.const.debug_file_suffix): + debug_files.append(pisi.util.join_path(root, f)) + + if not len(debug_files): + return None + + debug_package_obj = pisi.specfile.Package() + debug_package_obj.debug_package = True + debug_package_obj.name = self.spec.source.name + ctx.const.debug_name_suffix + # FIXME: find a better way to deal with the summary and description constants. + debug_package_obj.summary['en'] = u'Debug files for %s' % (self.spec.source.name) + debug_package_obj.description['en'] = u'Debug files for %s' % (self.spec.source.name) + debug_package_obj.partOf = self.spec.source.partOf + '-debug' + for f in debug_files: + debug_package_obj.files.append(pisi.specfile.Path(path = f[len(self.pkg_debug_dir()):], fileType = "debug")) + + # append all generated packages to dependencies + for p in self.spec.packages: + debug_package_obj.packageDependencies.append( + pisi.dependency.Dependency(package = p.name)) + + return debug_package_obj + + def strip_install_dir(self): + """strip install directory""" + ctx.ui.action(_("Stripping files..")) + install_dir = self.pkg_install_dir() + try: + nostrip = self.actionGlobals['NoStrip'] + pisi.util.strip_directory(install_dir, nostrip) + except KeyError: + pisi.util.strip_directory(install_dir) + + def gen_metadata_xml(self, package): + """Generate the metadata.xml file for build source. + + metadata.xml is composed of the information from specfile plus + some additional information.""" + metadata = pisi.metadata.MetaData() + metadata.from_spec(self.spec.source, package, self.spec.history) + + metadata.package.distribution = ctx.config.values.general.distribution + metadata.package.distributionRelease = ctx.config.values.general.distribution_release + metadata.package.architecture = "Any" + metadata.package.packageFormat = ctx.get_option('package_format') + + size = 0 + for fileinfo in self.files.list: + size += fileinfo.size + + metadata.package.installedSize = size + + self.metadata = metadata + + def gen_files_xml(self, package): + """Generates files.xml using the path definitions in specfile and + the files produced by the build system.""" + files = pisi.files.Files() + + if package.debug_package: + install_dir = self.pkg_debug_dir() + else: + install_dir = self.pkg_install_dir() + + # FIXME: We need to expand globs before trying to calculate hashes + # Not on the fly like now. + + # we'll exclude collisions in get_file_hashes. Having a + # collisions list is not wrong, we must just handle it :). + collisions = check_path_collision(package, self.spec.packages) + # FIXME: material collisions after expanding globs could be + # reported as errors + + d = {} + def add_path(path): + # add the files under material path + for fpath, fhash in pisi.util.get_file_hashes(path, collisions, install_dir): + if ctx.get_option('create_static') \ + and fpath.endswith(ctx.const.ar_file_suffix) \ + and not package.name.endswith(ctx.const.static_name_suffix) \ + and pisi.util.is_ar_file(fpath): + # if this is an ar file, and this package is not a static package, + # don't include this file into the package. + continue + frpath = pisi.util.removepathprefix(install_dir, fpath) # relative path + ftype, permanent = get_file_type(frpath, package.files, install_dir) + fsize = pisi.util.dir_size(fpath) + if not os.path.islink(fpath): + st = os.stat(fpath) + else: + st = os.lstat(fpath) + d[frpath] = pisi.files.FileInfo(path=frpath, type=ftype, permanent=permanent, + size=fsize, hash=fhash, uid=str(st.st_uid), gid=str(st.st_gid), + mode=oct(stat.S_IMODE(st.st_mode))) + + for pinfo in package.files: + wildcard_path = pisi.util.join_path(install_dir, pinfo.path) + for path in glob.glob(wildcard_path): + add_path(path) + + for (p, fileinfo) in d.iteritems(): + files.append(fileinfo) + + files_xml_path = pisi.util.join_path(self.pkg_dir(), ctx.const.files_xml) + files.write(files_xml_path) + self.files = files + + def calc_build_no(self, package_name): + """Calculate build number""" + + def metadata_changed(old_metadata, new_metadata): + for key in old_metadata.package.__dict__.keys(): + if old_metadata.package.__dict__[key] != new_metadata.package.__dict__[key]: + if key != "build": + return True + + return False + + # find previous build in packages dir + found = [] + def locate_old_package(old_package_fn): + if pisi.util.is_package_name(os.path.basename(old_package_fn), package_name): + try: + old_pkg = pisi.package.Package(old_package_fn, 'r') + old_pkg.read(pisi.util.join_path(ctx.config.tmp_dir(), 'oldpkg')) + ctx.ui.info(_('(found old version %s)') % old_package_fn) + if str(old_pkg.metadata.package.name) != package_name: + ctx.ui.warning(_('Skipping %s with wrong pkg name ') % + old_package_fn) + return + old_build = old_pkg.metadata.package.build + found.append( (old_package_fn, old_build) ) + except Error: + ctx.ui.warning('Package file %s may be corrupt. Skipping.' % old_package_fn) + + for root, dirs, files in os.walk(ctx.config.compiled_packages_dir()): + for f in files: + locate_old_package(pisi.util.join_path(root,f)) + + outdir=ctx.get_option('output_dir') + if not outdir: + outdir = '.' + for f in [pisi.util.join_path(outdir,entry) for entry in os.listdir(outdir)]: + if os.path.isfile(f): + locate_old_package(f) + + if not found: + return (1, None) + ctx.ui.warning(_('(no previous build found, setting build no to 1.)')) + else: + a = filter(lambda (x,y): y != None, found) + ctx.ui.debug(str(a)) + if a: + # sort in order of increasing build number + a.sort(lambda x,y : cmp(x[1],y[1])) + old_package_fn = a[-1][0] # get the last one + old_build = a[-1][1] + + # compare old files.xml with the new one.. + old_pkg = pisi.package.Package(old_package_fn, 'r') + old_pkg.read(pisi.util.join_path(ctx.config.tmp_dir(), 'oldpkg')) + + changed = False + fnew = self.files.list + fold = old_pkg.files.list + fold.sort(lambda x,y : cmp(x.path,y.path)) + fnew.sort(lambda x,y : cmp(x.path,y.path)) + + if len(fnew) != len(fold): + changed = True + else: + for i in range(len(fold)): + fo = fold.pop(0) + fn = fnew.pop(0) + if fo.path != fn.path: + changed = True + break + else: + if fo.hash != fn.hash: + changed = True + break + + if metadata_changed(old_pkg.metadata, self.metadata): + changed = True + + self.old_packages.append(os.path.basename(old_package_fn)) + else: # no old build had a build number + old_build = None + + ctx.ui.debug('old build number: %s' % old_build) + + # set build number + if old_build is None: + ctx.ui.warning(_('(old package lacks a build no, setting build no to 1.)')) + return (1, None) + elif changed: + ctx.ui.info(_('There are changes, incrementing build no to %d') % (old_build + 1)) + return (old_build + 1, old_build) + else: + ctx.ui.info(_('There is no change from previous build %d') % old_build) + return (old_build, old_build) + + def build_packages(self): + """Build each package defined in PSPEC file. After this process there + will be .pisi files hanging around, AS INTENDED ;)""" + + self.fetch_component() # bug 856 + + # Strip install directory before building .pisi packages. + self.strip_install_dir() + + if ctx.get_option('create_static'): + obj = self.generate_static_package_object() + if obj: + self.spec.packages.append(obj) + + if ctx.config.values.build.generatedebug: + obj = self.generate_debug_package_object() + if obj: + self.spec.packages.append(obj) + + self.new_packages = [] + self.old_packages = [] + + for package in self.spec.packages: + + # removing "farce" in specfile.py:SpecFile.override_tags + # this block of code came here... SpecFile should never + # ever ruin the generated PSPEC file. If build process + # needs this, we should do it in here... (bug: #3773) + if not package.summary: + package.summary = self.spec.source.summary + if not package.description: + # TODO: remove this if statement with the part in + # specfile.py:SpecFile + if not self.spec.source.description: + self.spec.dirtyWorkAround() + + package.description = self.spec.source.description + if not package.partOf: + package.partOf = self.spec.source.partOf + if not package.license: + package.license = self.spec.source.license + if not package.icon: + package.icon = self.spec.source.icon + + + + old_package_name = None + # store additional files + c = os.getcwd() + os.chdir(self.specdir) + install_dir = self.pkg_dir() + ctx.const.install_dir_suffix + for afile in package.additionalFiles: + src = os.path.join(ctx.const.files_dir, afile.filename) + dest = os.path.join(install_dir + os.path.dirname(afile.target), os.path.basename(afile.target)) + pisi.util.copy_file(src, dest) + if afile.permission: + # mode is octal! + os.chmod(dest, int(afile.permission, 8)) + if afile.owner: + try: + os.chown(dest, pwd.getpwnam(afile.owner)[2], -1) + except KeyError: + ctx.ui.warning(_("No user named '%s' found on the system") % afile.owner) + if afile.group: + try: + os.chown(dest, -1, grp.getgrnam(afile.group)[2]) + except KeyError: + ctx.ui.warning(_("No group named '%s' found on the system") % afile.group) + os.chdir(c) + + ctx.ui.action(_("** Building package %s") % package.name); + + ctx.ui.info(_("Generating %s,") % ctx.const.files_xml) + self.gen_files_xml(package) + + ctx.ui.info(_("Generating %s,") % ctx.const.metadata_xml) + self.gen_metadata_xml(package) + + # build number + if ctx.config.options.ignore_build_no or not ctx.config.values.build.buildno: + build_no = old_build_no = None + ctx.ui.warning(_('Build number is not available. For repo builds you must enable buildno in pisi.conf.')) + else: + build_no, old_build_no = self.calc_build_no(package.name) + + self.metadata.package.build = build_no + self.metadata.write(pisi.util.join_path(self.pkg_dir(), ctx.const.metadata_xml)) + + # Calculate new and oldpackage names for buildfarm + name = pisi.util.package_name(package.name, + self.spec.getSourceVersion(), + self.spec.getSourceRelease(), + self.metadata.package.build) + + outdir = ctx.get_option('output_dir') + if outdir: + name = pisi.util.join_path(outdir, name) + self.new_packages.append(name) + + ctx.ui.info(_("Creating PiSi package %s.") % name) + + pkg = pisi.package.Package(name, 'w') + + # add comar files to package + os.chdir(self.specdir) + for pcomar in package.providesComar: + fname = pisi.util.join_path(ctx.const.comar_dir, + pcomar.script) + pkg.add_to_package(fname) + + # add xmls and files + os.chdir(self.pkg_dir()) + + pkg.add_to_package(ctx.const.metadata_xml) + pkg.add_to_package(ctx.const.files_xml) + + # Now it is time to add files to the packages using newly + # created files.xml + files = pisi.files.Files() + files.read(ctx.const.files_xml) + + if ctx.get_option('package_format') == "1.0": + for finfo in files.list: + orgname = arcname = pisi.util.join_path("install", finfo.path) + if package.debug_package: + orgname = pisi.util.join_path("debug", finfo.path) + pkg.add_to_package(orgname, arcname) + pkg.close() + else: # default package format is 1.1, so make it fallback. + ctx.build_leftover = pisi.util.join_path(self.pkg_dir(), ctx.const.install_tar_lzma) + tar = archive.ArchiveTar(ctx.const.install_tar_lzma, "tarlzma") + for finfo in files.list: + orgname = arcname = pisi.util.join_path("install", finfo.path) + if package.debug_package: + orgname = pisi.util.join_path("debug", finfo.path) + tar.add_to_archive(orgname, arcname.lstrip("install")) + tar.close() + pkg.add_to_package(ctx.const.install_tar_lzma) + pkg.close() + os.unlink(ctx.const.install_tar_lzma) + ctx.build_leftover = None + + os.chdir(c) + self.set_state("buildpackages") + ctx.ui.info(_("Done.")) + + #show the files those are not collected from the install dir + if ctx.get_option('debug'): + abandoned_files = self.get_abandoned_files() + if abandoned_files: + ctx.ui.warning(_('Abandoned files under the install dir (%s):') % (install_dir)) + for f in abandoned_files: + ctx.ui.info(' - %s' % (f)) + else: + ctx.ui.warning(_('All of the files under the install dir (%s) has been collected by package(s)') + % (install_dir)) + + if ctx.config.values.general.autoclean is True: + ctx.ui.info(_("Cleaning Build Directory...")) + pisi.util.clean_dir(self.pkg_dir()) + else: + ctx.ui.info(_("Keeping Build Directory")) + + + # reset environment variables after build. this one is for + # buildfarm actually. buildfarm re-inits pisi for each build + # and left environment variables go directly into initial dict + # making actionsapi.variables.exportFlags() useless... + os.environ = {} + os.environ = copy.deepcopy(ctx.config.environ) + + return self.new_packages, self.old_packages + + +# build functions... + +def build(pspec): + if pspec.endswith('.xml'): + pb = Builder(pspec) + else: + pb = Builder.from_name(pspec) + return pb.build() + +order = {"none": 0, + "fetch": 1, + "unpack": 2, + "setupaction": 3, + "buildaction": 4, + "installaction": 5, + "buildpackages": 6} + +def __buildState_fetch(pb): + # fetch is the first state to run. + pb.patch_exists() + pb.fetch_source_archive() + +def __buildState_unpack(pb, last): + if order[last] < order["fetch"]: + __buildState_fetch(pb) + pb.unpack_source_archive() + +def __buildState_setupaction(pb, last): + if order[last] < order["unpack"]: + __buildState_unpack(pb, last) + pb.run_setup_action() + +def __buildState_buildaction(pb, last): + + if order[last] < order["setupaction"]: + __buildState_setupaction(pb, last) + pb.run_build_action() + +def __buildState_checkaction(pb, last): + + if order[last] < order["buildaction"]: + __buildState_buildaction(pb, last) + pb.run_check_action() + +def __buildState_installaction(pb, last): + + if order[last] < order["buildaction"]: + __buildState_buildaction(pb, last) + pb.run_install_action() + +def __buildState_buildpackages(pb, last): + + if order[last] < order["installaction"]: + __buildState_installaction(pb, last) + pb.build_packages() + +def build_until(pspec, state): + if pspec.endswith('.xml'): + pb = Builder(pspec) + else: + pb = Builder.from_name(pspec) + + pb.compile_action_script() + pb.compile_comar_script() + + last = pb.get_state() + ctx.ui.info("Last state was %s"%last) + + if not last: last = "none" + + if state == "fetch": + __buildState_fetch(pb) + return + + if state == "unpack": + __buildState_unpack(pb, last) + return + + # from now on build dependencies are needed + pb.check_build_dependencies() + + if state == "setup": + __buildState_setupaction(pb, last) + return + + if state == "build": + __buildState_buildaction(pb, last) + return + + if state == "check": + __buildState_checkaction(pb, last) + return + + if state == "install": + __buildState_installaction(pb, last) + return + + __buildState_buildpackages(pb, last) diff --git a/branches/pisi-1.1/pisi/cli/README b/branches/pisi-1.1/pisi/cli/README new file mode 100644 index 00000000..5ad52a09 --- /dev/null +++ b/branches/pisi-1.1/pisi/cli/README @@ -0,0 +1,23 @@ +Writing commands: +----------------- + +TODO: this is a *bit* out of date now. + +Subclass from Command. + +Be careful not to import PiSi modules before Command is run, e.g. +do it in run() method. This is necessary to prevent an obscure +initialization error. + +We used to import pisi.operations in pisi.cli + +This prevented pisi.operations from having the current global +pisi.ui.ui because it's loaded before the UI is set! So, despite +what you might understand from the python FAQ 1.2.3 +http://www.python.org/doc/faq/programming.html#how-do-i-share-global-variables-across-modules +you have to be careful about the initialization order when using +singleton modules. + +Another way to solve this is to ensure that the global module is +always imported again in every function it is called (see +ui.CLI.confirm() ) but this is more expensive. diff --git a/branches/pisi-1.1/pisi/cli/__init__.py b/branches/pisi-1.1/pisi/cli/__init__.py new file mode 100644 index 00000000..a0efcbc3 --- /dev/null +++ b/branches/pisi-1.1/pisi/cli/__init__.py @@ -0,0 +1,162 @@ +# -*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. +# + +import sys +import logging +import locale + +import gettext +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +import pisi +import pisi.context as ctx +import pisi.ui +import pisi.util + +class Error(pisi.Error): + pass + + +class Exception(pisi.Exception): + pass + + +def printu(obj, err = False): + if not isinstance(obj, unicode): + obj = unicode(obj) + if err: + out = sys.stderr + else: + out = sys.stdout + out.write(obj.encode('utf-8')) + out.flush() + +class CLI(pisi.ui.UI): + "Command Line Interface" + + def __init__(self, show_debug = False, show_verbose = False): + super(CLI, self).__init__(show_debug, show_verbose) + + def close(self): + pisi.util.xterm_title_reset() + + def output(self, msg, err = False, verbose = False): + if (verbose and self.show_verbose) or (not verbose): + if type(msg)==type(unicode()): + msg = msg.encode('utf-8') + if err: + out = sys.stderr + else: + out = sys.stdout + out.write(msg) + out.flush() + + def info(self, msg, verbose = False, noln = False): + # TODO: need to look at more kinds of info messages + # let's cheat from KDE :) + if not noln: + msg = '%s\n' % msg + self.output(unicode(msg), verbose=verbose) + + def warning(self, msg, verbose = False): + msg = unicode(msg) + if ctx.log: + ctx.log.warning(msg) + if ctx.get_option('no_color'): + self.output(_('Warning: ') + msg + '\n', err=True, verbose=verbose) + else: + self.output(pisi.util.colorize(msg + '\n', 'brightred'), err=True, verbose=verbose) + + def error(self, msg): + msg = unicode(msg) + if ctx.log: + ctx.log.error(msg) + if ctx.get_option('no_color'): + self.output(_('Error: ') + msg + '\n', err=True) + else: + self.output(pisi.util.colorize(msg + '\n', 'red'), err=True) + + def action(self, msg, verbose = False): + #TODO: this seems quite redundant? + msg = unicode(msg) + if ctx.log: + ctx.log.info(msg) + self.output(pisi.util.colorize(msg + '\n', 'green')) + + def choose(self, msg, opts): + print msg + for i in range(0,len(opts)): + print i + 1, opts(i) + while True: + s = raw_input(msg + pisi.util.colorize('1-%d' % len(opts), 'red')) + try: + opt = int(s) + if 1 <= opt and opt <= len(opts): + return opts(opt-1) + except Exception: + pass + + def confirm(self, msg): + msg = unicode(msg) + if ctx.config.options and ctx.config.options.yes_all: + return True + while True: + import re + yesexpr = re.compile(locale.nl_langinfo(locale.YESEXPR)) + + prompt = msg + pisi.util.colorize(_(' (yes/no)'), 'red') + s = raw_input(prompt.encode('utf-8')) + if yesexpr.search(s): + return True + + return False + + def display_progress(self, operation, percent, info="", **ka): + """ display progress of any operation """ + if operation in ["removing", "rebuilding-db"]: + return + elif operation == "fetching": + totalsize = '%.1f %s' % pisi.util.human_readable_size(ka['total_size']) + out = '\r%-30.30s (%s)%3d%% %9.2f %s [%s]' % \ + (ka['filename'], totalsize, percent, + ka['rate'], ka['symbol'], ka['eta']) + self.output(out) + else: + self.output("\r%s (%d%%)" % (info, percent)) + + if percent == 100: + self.output(pisi.util.colorize(_(' [complete]\n'), 'gray')) + + def status(self, msg = None): + if msg: + msg = unicode(msg) + self.output(pisi.util.colorize(msg + '\n', 'brightgreen')) + pisi.util.xterm_title(msg) + + def notify(self, event, **keywords): + if event == pisi.ui.installed: + msg = _('Installed %s') % keywords['package'].name + elif event == pisi.ui.removed: + msg = _('Removed %s') % keywords['package'].name + elif event == pisi.ui.upgraded: + msg = _('Upgraded %s') % keywords['package'].name + elif event == pisi.ui.configured: + msg = _('Configured %s') % keywords['package'].name + elif event == pisi.ui.extracting: + msg = _('Extracting the files of %s') % keywords['package'].name + else: + msg = None + if msg: + self.output(pisi.util.colorize(msg + '\n', 'cyan')) + if ctx.log: + ctx.log.info(msg) diff --git a/branches/pisi-1.1/pisi/cli/commands.py b/branches/pisi-1.1/pisi/cli/commands.py new file mode 100644 index 00000000..c2c19a79 --- /dev/null +++ b/branches/pisi-1.1/pisi/cli/commands.py @@ -0,0 +1,1685 @@ +# -*- coding:utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. +# + +import sys +from optparse import OptionParser, OptionGroup, HelpFormatter + +import gettext +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +import pisi +import pisi.cli +import pisi.context as ctx +from pisi.uri import URI +import pisi.util as util + +class Error(pisi.Error): + pass + +class PisiHelpFormatter(HelpFormatter): + def __init__(self, + indent_increment=1, + max_help_position=32, + width=None, + short_first=1): + HelpFormatter.__init__( + self, indent_increment, max_help_position, width, short_first) + + self._short_opt_fmt = "%s" + self._long_opt_fmt = "%s" + + def format_usage(self, usage): + return _("usage: %s\n") % usage + + def format_heading(self, heading): + return "%*s%s:\n" % (self.current_indent, "", heading) + + def format_option_strings(self, option): + """Return a comma-separated list of option strings & metavariables.""" + if option.takes_value(): + short_opts = [self._short_opt_fmt % sopt + for sopt in option._short_opts] + long_opts = [self._long_opt_fmt % lopt + for lopt in option._long_opts] + else: + short_opts = option._short_opts + long_opts = option._long_opts + + if long_opts and short_opts: + opt = "%s [%s]" % (short_opts[0], long_opts[0]) + else: + opt = long_opts[0] or short_opts[0] + + if option.takes_value(): + opt += " arg" + + return opt + + def format_option(self, option): + import textwrap + result = [] + opts = self.option_strings[option] + opt_width = self.help_position - self.current_indent - 2 + if len(opts) > opt_width: + opts = "%*s%s\n" % (self.current_indent, "", opts) + indent_first = self.help_position + else: # start help on same line as opts + opts = "%*s%-*s " % (self.current_indent, "", opt_width, opts) + indent_first = 0 + result.append(opts) + if option.help: + help_text = self.expand_default(option) + help_lines = textwrap.wrap(help_text, self.help_width) + result.append(": %*s%s\n" % (indent_first, "", help_lines[0])) + result.extend([" %*s%s\n" % (self.help_position, "", line) + for line in help_lines[1:]]) + elif opts[-1] != "\n": + result.append("\n") + return "".join(result) + + +class Command(object): + """generic help string for any command""" + + # class variables + + cmd = [] + cmd_dict = {} + + @staticmethod + def commands_string(): + s = '' + l = [x.name[0] for x in Command.cmd] + l.sort() + for name in l: + commandcls = Command.cmd_dict[name] + trans = gettext.translation('pisi', fallback=True) + summary = trans.ugettext(commandcls.__doc__).split('\n')[0] + name = commandcls.name[0] + if commandcls.name[1]: + name += ' (%s)' % commandcls.name[1] + s += '%21s - %s\n' % (name, summary) + return s + + @staticmethod + def get_command(cmd, fail=False, args=None): + + if Command.cmd_dict.has_key(cmd): + return Command.cmd_dict[cmd](args) + + if fail: + raise Error(_("Unrecognized command: %s") % cmd) + else: + return None + + # instance variabes + + def __init__(self, args = None): + # now for the real parser + import pisi + self.comar = False + self.parser = OptionParser(usage=getattr(self, "__doc__"), + version="%prog " + pisi.__version__, + formatter=PisiHelpFormatter()) + self.options() + self.commonopts() + (self.options, self.args) = self.parser.parse_args(args) + if self.args: + self.args.pop(0) # exclude command arg + + self.process_opts() + + def commonopts(self): + '''common options''' + p = self.parser + + group = OptionGroup(self.parser, _("general options")) + + group.add_option("-D", "--destdir", action="store", default = None, + help = _("Change the system root for PiSi commands")) + group.add_option("-y", "--yes-all", action="store_true", + default=False, help = _("Assume yes in all yes/no queries")) + group.add_option("-u", "--username", action="store") + group.add_option("-p", "--password", action="store") + group.add_option("-v", "--verbose", action="store_true", + dest="verbose", default=False, + help=_("Detailed output")) + group.add_option("-d", "--debug", action="store_true", + default=False, help=_("Show debugging information")) + group.add_option("-N", "--no-color", action="store_true", default=False, + help = _("Suppresses all coloring of PiSi's output")) + + p.add_option_group(group) + + return p + + def options(self): + """This is a fall back function. If the implementer module provides an + options function it will be called""" + pass + + def process_opts(self): + self.check_auth_info() + + # make destdir absolute + if self.options.destdir: + d = str(self.options.destdir) + import os.path + if not os.path.exists(d): + pisi.cli.printu(_('Destination directory %s does not exist. Creating directory.\n') % d) + os.makedirs(d) + self.options.destdir = os.path.realpath(d) + + def check_auth_info(self): + username = self.options.username + password = self.options.password + + # TODO: We'll get the username, password pair from a configuration + # file from users home directory. Currently we need user to + # give it from the user interface. + # if not username and not password: + # if someauthconfig.username and someauthconfig.password: + # self.authInfo = (someauthconfig.username, + # someauthconfig.password) + # return + if username and password: + self.options.authinfo = (username, password) + return + + if username and not password: + from getpass import getpass + password = getpass(_("Password: ")) + self.options.authinfo = (username, password) + else: + self.options.authinfo = None + + def init(self, database = True, write = True): + """initialize PiSi components""" + + # NB: command imports here or in the command class run fxns + import pisi.api + pisi.api.init(database = database, write = write, options = self.options, + comar = self.comar) + + def finalize(self): + """do cleanup work for PiSi components""" + pisi.api.finalize() + + def get_name(self): + return self.__class__.name + + def format_name(self): + (name, shortname) = self.get_name() + if shortname: + return "%s (%s)" % (name, shortname) + else: + return name + + def help(self): + """print help for the command""" + trans = gettext.translation('pisi', fallback=True) + print "%s: %s\n" % (self.format_name(), trans.ugettext(self.__doc__)) + print self.parser.format_option_help() + + def die(self): + """exit program""" + #FIXME: not called from anywhere? + ctx.ui.error(_('Command terminated abnormally.')) + sys.exit(-1) + + +class autocommand(type): + def __init__(cls, name, bases, dict): + super(autocommand, cls).__init__(name, bases, dict) + Command.cmd.append(cls) + name = getattr(cls, 'name', None) + if name is None: + raise pisi.cli.Error(_('Command lacks name')) + longname, shortname = name + def add_cmd(cmd): + if Command.cmd_dict.has_key(cmd): + raise pisi.cli.Error(_('Duplicate command %s') % cmd) + else: + Command.cmd_dict[cmd] = cls + add_cmd(longname) + if shortname: + add_cmd(shortname) + + +class Help(Command): + """Prints help for given commands + +Usage: help [ ... ] + +If run without parameters, it prints the general help.""" + + __metaclass__ = autocommand + + def __init__(self, args = None): + #TODO? Discard Help's own usage doc in favor of general usage doc + #self.__doc__ = usage_text + #self.__doc__ += commands_string() + super(Help, self).__init__(args) + + name = ("help", "?") + + def run(self): + + if not self.args: + self.parser.set_usage(usage_text) + pisi.cli.printu(self.parser.format_help()) + return + + self.init(database = False, write = False) + + for arg in self.args: + obj = Command.get_command(arg, True) + obj.help() + ctx.ui.info('') + + self.finalize() + + +class Clean(Command): + """Clean stale locks + +Usage: clean + +PiSi uses filesystem locks for managing database access. +This command deletes unused locks from the database directory.""" + + __metaclass__ = autocommand + + def __init__(self, args=None): + super(Clean, self).__init__(args) + + name = ("clean", None) + + def run(self): + self.init() + self.finalize() + +class DeleteCache(Command): + """Delete cache files + +Usage: delete-cache + +Sources, packages and temporary files are stored +under /var directory. Since these accumulate they can +consume a lot of disk space.""" + + __metaclass__ = autocommand + + def __init__(self, args=None): + super(DeleteCache, self).__init__(args) + + name = ("delete-cache", "dc") + + def run(self): + self.init(database=False, write=True) + pisi.api.delete_cache() + + +class Graph(Command): + """Graph package relations + +Usage: graph [ ...] + +Write a graph of package relations, tracking dependency and +conflicts relations starting from given packages. By default +shows the package relations among repository packages, and writes +the package in graphviz format to 'pgraph.dot'. +""" + + __metaclass__ = autocommand + + def __init__(self, args=None): + super(Graph, self).__init__(args) + + def options(self): + + group = OptionGroup(self.parser, _("graph options")) + + group.add_option("-r", "--repository", action="store", + default=None, + help=_("Specify a particular repository")) + group.add_option("-i", "--installed", action="store_true", + default=False, + help=_("Graph of installed packages")) + group.add_option("--ignore-installed", action="store_true", + default=False, + help=_("Do not show installed packages")) + group.add_option("-o", "--output", action="store", + default='pgraph.dot', + help=_("Dot output file")) + + self.parser.add_option_group(group) + + name = ("graph", None) + + def run(self): + self.init(write=False) + if not ctx.get_option('installed'): + if ctx.get_option('repository'): + repo = ctx.get_option('repository') + ctx.ui.info(_('Plotting packages in repository %s') % repo) + else: + repo = pisi.itembyrepodb.repos + if self.args: + a = self.args + else: + ctx.ui.info(_('Plotting a graph of relations among all repository packages')) + a = ctx.packagedb.list_packages(repo) + else: + if self.args: + a = self.args + else: + # if A is empty, then graph all packages + ctx.ui.info(_('Plotting a graph of relations among all installed packages')) + a = ctx.installdb.list_installed() + repo = pisi.itembyrepodb.installed + g = pisi.api.package_graph(a, repo = repo, + ignore_installed = ctx.get_option('ignore_installed')) + g.write_graphviz(file(ctx.get_option('output'), 'w')) + self.finalize() + +# option mixins +def buildno_opts(self, group): + group.add_option("--ignore-build-no", action="store_true", + default=False, + help=_("Do not take build no into account.")) + +def ignoredep_opt(self, group): + group.add_option("--ignore-dependency", action="store_true", + default=False, + help=_("Do not take dependency information into account")) + + +class Build(Command): + """Build PiSi packages + +Usage: build [ | ] ... + +You can give a URI of the pspec.xml file. PiSi will +fetch all necessary files and build the package for you. + +Alternatively, you can give the name of a source package +to be downloaded from a repository containing sources. +""" + __metaclass__ = autocommand + + def __init__(self, args): + super(Build, self).__init__(args) + self.comar = True + + name = ("build", "bi") + + package_formats = ('1.0', '1.1') + + def options(self): + + self.add_steps_options() + group = OptionGroup(self.parser, _("build options")) + self.add_options(group) + self.parser.add_option_group(group) + + def add_options(self, group): + buildno_opts(self, group) + ignoredep_opt(self, group) + group.add_option("-O", "--output-dir", action="store", default=None, + help=_("Output directory for produced packages")) + group.add_option("--ignore-action-errors", + action="store_true", default=False, + help=_("Bypass errors from ActionsAPI")) + group.add_option("--ignore-safety", action="store_true", + default=False, help=_("Bypass safety switch")) + group.add_option("--ignore-check", action="store_true", + default=False, help=_("Bypass testing step")) + group.add_option("--create-static", action="store_true", + default=False, help=_("Create a static package with ar files")) + group.add_option("--no-install", action="store_true", + default=False, help=_("Do not install build dependencies, fail if a build dependency is present")) + group.add_option("-F", "--package-format", action="store", default='1.1', + help=_("PiSi binary package formats: '1.0', '1.1' (default)")) + group.add_option("--use-quilt", action="store_true", default=False, + help=_("Use quilt patch management system instead of GNU patch")) + group.add_option("--ignore-sandbox", action="store_true", default=False, + help=_("Do not constrain build process inside the build folder")) + + def add_steps_options(self): + group = OptionGroup(self.parser, _("build steps")) + group.add_option("--fetch", dest="until", action="store_const", + const="fetch", help=_("Break build after fetching the source archive")) + group.add_option("--unpack", dest="until", action="store_const", + const="unpack", help=_("Break build after unpacking the source archive, checking sha1sum and applying patches")) + group.add_option("--setup", dest="until", action="store_const", + const="setup", help=_("Break build after running configure step")) + group.add_option("--build", dest="until", action="store_const", + const="build", help=_("Break build after running compile step")) + group.add_option("--check", dest="until", action="store_const", + const="check", help=_("Break build after running check step")) + group.add_option("--install", dest="until", action="store_const", + const="install", help=_("Break build after running install step")) + group.add_option("--package", dest="until", action="store_const", + const="package", help=_("create PiSi package")) + self.parser.add_option_group(group) + + def run(self): + if not self.args: + self.help() + return + + if self.options.no_install: + self.init(database=True, write=False) + else: + self.init() + + if ctx.get_option('package_format') not in Build.package_formats: + raise Error(_('package_format must be one of %s ') % pisi.util.strlist(Build.package_formats)) + + if ctx.get_option('output_dir'): + ctx.ui.info(_('Output directory: %s') % ctx.config.options.output_dir) + else: + ctx.ui.info(_('Outputting packages in the working directory.')) + ctx.config.options.output_dir = '.' + + for x in self.args: + if ctx.get_option('until'): + pisi.api.build_until(x, ctx.get_option('until')) + else: + pisi.api.build(x) + self.finalize() + +class Delta(Command): + """Creates delta PiSi packages + +Usage: delta oldpackage newpackage + +Delta command finds the changed files between the given packages by comparing the sha1sum of the files +and creates a delta pisi package with the changed files between two releases. + +""" + __metaclass__ = autocommand + + def __init__(self, args): + super(Delta, self).__init__(args) + + name = ("delta", "dt") + + def options(self): + + group = OptionGroup(self.parser, _("delta options")) + self.add_options(group) + self.parser.add_option_group(group) + + def add_options(self, group): + group.add_option("-O", "--output-dir", action="store", default=None, + help=_("Output directory for produced packages")) + + def run(self): + + from pisi.delta import create_delta_package + + self.init(database=False, write=False) + + if len(self.args) != 2: + self.help() + return + + if ctx.get_option('output_dir'): + ctx.ui.info(_('Output directory: %s') % ctx.config.options.output_dir) + else: + ctx.ui.info(_('Outputting packages in the working directory.')) + ctx.config.options.output_dir = '.' + + oldpackage = self.args[0] + newpackage = self.args[1] + + create_delta_package(oldpackage, newpackage) + + self.finalize() + +class Emerge(Build): + """Build and install PiSi source packages from repository + +Usage: emerge ... + +You should give the name of a source package to be +downloaded from a repository containing sources. + +You can also give the name of a component. +""" + __metaclass__ = autocommand + + def __init__(self, args): + super(Emerge, self).__init__(args) + self.comar = True + + name = ("emerge", "em") + + def options(self): + + group = OptionGroup(self.parser, _("emerge options")) + super(Emerge, self).add_options(group) + group.add_option("--ignore-file-conflicts", action="store_true", + default=False, help=_("Ignore file conflicts")) + group.add_option("--ignore-package-conflicts", action="store_true", + default=False, help=_("Ignore package conflicts")) + group.add_option("--ignore-comar", action="store_true", + default=False, help=_("Bypass comar configuration agent")) + self.parser.add_option_group(group) + + def run(self): + if not self.args: + self.help() + return + + self.init(database = True) + if ctx.get_option('output_dir'): + ctx.ui.info(_('Output directory: %s') % ctx.config.options.output_dir) + else: + ctx.ui.info(_('Outputting binary packages in the package cache.')) + ctx.config.options.output_dir = ctx.config.packages_dir() + + pisi.api.emerge(self.args) + self.finalize() + + +class PackageOp(Command): + """Abstract package operation command""" + + def __init__(self, args): + super(PackageOp, self).__init__(args) + self.comar = True + + def options(self, group): + ignoredep_opt(self, group) + group.add_option("--ignore-comar", action="store_true", + default=False, help=_("Bypass comar configuration agent")) + group.add_option("--ignore-safety", action="store_true", + default=False, help=_("Bypass safety switch")) + group.add_option("-n", "--dry-run", action="store_true", default=False, + help = _("Do not perform any action, just show what would be done")) + + def init(self, database=True, write=True): + super(PackageOp, self).init(database, write) + + def finalize(self): + #self.finalize_db() + pass + + +class Install(PackageOp): + """Install PiSi packages + +Usage: install ... + +You may use filenames, URI's or package names for packages. If you have +specified a package name, it should exist in a specified repository. + +You can also specify components instead of package names, which will be +expanded to package names. +""" + __metaclass__ = autocommand + + def __init__(self, args): + super(Install, self).__init__(args) + + name = "install", "it" + + def options(self): + group = OptionGroup(self.parser, _("install options")) + + super(Install, self).options(group) + buildno_opts(self, group) + group.add_option("--reinstall", action="store_true", + default=False, help=_("Reinstall already installed packages")) + group.add_option("--ignore-file-conflicts", action="store_true", + default=False, help=_("Ignore file conflicts")) + group.add_option("--ignore-package-conflicts", action="store_true", + default=False, help=_("Ignore package conflicts")) + group.add_option("-c", "--component", action="append", + default=None, help=_("Install component's and recursive components' packages")) + group.add_option("-f", "--fetch-only", action="store_true", + default=False, help=_("Fetch upgrades but do not install.")) + self.parser.add_option_group(group) + + def run(self): + + if self.options.fetch_only: + self.init(database=True, write=False) + else: + self.init() + + components = ctx.get_option('component') + if not components and not self.args: + self.help() + return + + packages = [] + if components: + for name in components: + if ctx.componentdb.has_component(name): + packages.extend(ctx.componentdb.get_union_packages(name, walk=True)) + packages.extend(self.args) + + pisi.api.install(packages, ctx.get_option('reinstall')) + self.finalize() + + +class Upgrade(PackageOp): + """Upgrade PiSi packages + +Usage: Upgrade [ ... ] + +: package name + +Upgrades the entire system if no package names are given + +You may use only package names to specify packages because +the package upgrade operation is defined only with respect +to repositories. If you have specified a package name, it +should exist in the package repositories. If you just want to +reinstall a package from a PiSi file, use the install command. + +You can also specify components instead of package names, which will be +expanded to package names. +""" + + __metaclass__ = autocommand + + def __init__(self, args): + super(Upgrade, self).__init__(args) + + name = ("upgrade", "up") + + def options(self): + group = OptionGroup(self.parser, _("upgrade options")) + + super(Upgrade, self).options(group) + buildno_opts(self, group) + group.add_option("--security-only", action="store_true", + default=False, help=_("Security related package upgrades only")) + group.add_option("-r", "--bypass-update-repo", action="store_true", + default=False, help=_("Do not update repositories")) + group.add_option("--ignore-file-conflicts", action="store_true", + default=False, help=_("Ignore file conflicts")) + group.add_option("--ignore-package-conflicts", action="store_true", + default=False, help=_("Ignore package conflicts")) + group.add_option("-c", "--component", action="append", + default=None, help=_("Upgrade component's and recursive components' packages")) + group.add_option("-f", "--fetch-only", action="store_true", + default=False, help=_("Fetch upgrades but do not install.")) + group.add_option("-x", "--exclude", action="append", + default=None, help=_("When upgrading system, ignore packages and components whose basenames match pattern.")) + group.add_option("--exclude-from", action="store", + default=None, help=_("When upgrading system, ignore packages and components whose basenames \ + match any pattern contained in file.")) + + self.parser.add_option_group(group) + + def exclude_from(self, packages): + import os + + patterns = [] + f = ctx.get_option('exclude_from') + if os.path.exists(f): + for line in open(f, "r").readlines(): + if not line.startswith('#') and not line == '\n': + patterns.append(line.strip()) + if patterns: + return self.exclude(packages, patterns) + + return packages + + def exclude(self, packages, patterns): + from sets import Set as set + import fnmatch + + packages = set(packages) + for pattern in patterns: + # match pattern in package names + match = fnmatch.filter(packages, pattern) + packages = packages - set(match) + + if not match: + # match pattern in component names + for compare in fnmatch.filter(ctx.componentdb.list_components(), pattern): + packages = packages - set(ctx.componentdb.get_union_packages(compare, walk=True)) + + return list(packages) + + def run(self): + + self.init() + + if not ctx.get_option('bypass_update_repo'): + ctx.ui.info(_('Updating repositories')) + repos = ctx.repodb.list() + for repo in repos: + pisi.api.update_repo(repo) + else: + ctx.ui.info(_('Will not update repositories')) + + components = ctx.get_option('component') + packages = [] + if components: + for name in components: + if ctx.componentdb.has_component(name): + packages.extend(ctx.componentdb.get_union_packages(name, walk=True)) + packages.extend(self.args) + + if packages == []: + packages = ctx.installdb.list_installed() + + if ctx.get_option('exclude_from'): + packages = self.exclude_from(packages) + + if ctx.get_option('exclude'): + patterns = ctx.get_option('exclude') + packages = self.exclude(packages, patterns) + + pisi.api.upgrade(packages) + self.finalize() + + +class Remove(PackageOp): + """Remove PiSi packages + +Usage: remove ... + +Remove package(s) from your system. Just give the package names to remove. + +You can also specify components instead of package names, which will be +expanded to package names. +""" + __metaclass__ = autocommand + + def __init__(self, args): + super(Remove, self).__init__(args) + + name = ("remove", "rm") + + def options(self): + group = OptionGroup(self.parser, _("remove options")) + super(Remove, self).options(group) + group.add_option("-c", "--component", action="append", + default=None, help=_("Remove component's and recursive components' packages")) + self.parser.add_option_group(group) + + def run(self): + self.init() + + components = ctx.get_option('component') + if not components and not self.args: + self.help() + return + + packages = [] + if components: + for name in components: + if ctx.componentdb.has_component(name): + packages.extend(ctx.componentdb.get_union_packages(name, walk=True)) + packages.extend(self.args) + + pisi.api.remove(packages) + self.finalize() + +class ConfigurePending(PackageOp): + """Configure pending packages + +If COMAR configuration of some packages were not +done at installation time, they are added to a list +of packages waiting to be configured. This command +configures those packages. +""" + + __metaclass__ = autocommand + + def __init__(self, args): + super(ConfigurePending, self).__init__(args) + + name = ("configure-pending", "cp") + + def options(self): + group = OptionGroup(self.parser, _("configure-pending options")) + super(ConfigurePending, self).options(group) + self.parser.add_option_group(group) + + def run(self): + + self.init() + pisi.api.configure_pending(self.args) + self.finalize() + + +class Info(Command): + """Display package information + +Usage: info ... + + is either a package name or a .pisi file, +""" + __metaclass__ = autocommand + + def __init__(self, args): + super(Info, self).__init__(args) + + name = ("info", None) + + def options(self): + + group = OptionGroup(self.parser, _("info options")) + self.add_options(group) + self.parser.add_option_group(group) + + def add_options(self, group): + group.add_option("-f", "--files", action="store_true", + default=False, + help=_("Show a list of package files.")) + group.add_option("-c", "--component", action="append", + default=None, help=_("Info about the given component")) + group.add_option("-F", "--files-path", action="store_true", + default=False, + help=_("Show only paths.")) + group.add_option("-s", "--short", action="store_true", + default=False, help=_("Do not show details")) + group.add_option("--xml", action="store_true", + default=False, help=_("Output in xml format")) + + def run(self): + + self.init(database = True, write = False) + + components = ctx.get_option('component') + if not components and not self.args: + self.help() + return + + index = pisi.index.Index() + index.distribution = None + + # info of components + if components: + for name in components: + if ctx.componentdb.has_component(name): + component = ctx.componentdb.get_union_comp(name) + if self.options.xml: + index.add_component(component) + else: + if not self.options.short: + ctx.ui.info(unicode(component)) + else: + ctx.ui.info("%s - %s" % (component.name, component.summary)) + + # info of packages + for arg in self.args: + if self.options.xml: + index.packages.append(pisi.api.info(arg)[0].package) + else: + self.info_package(arg) + + if self.options.xml: + errs = [] + index.newDocument() + index.encode(index.rootNode(), errs) + index.writexmlfile(sys.stdout) + sys.stdout.write('\n') + self.finalize() + + + def info_package(self, arg): + if arg.endswith(ctx.const.package_suffix): + metadata, files = pisi.api.info_file(arg) + ctx.ui.info(_('Package file: %s') % arg) + self.print_pkginfo(metadata, files) + else: + if ctx.installdb.is_installed(arg): + metadata, files, repo = pisi.api.info_name(arg, True) + if self.options.short: + ctx.ui.info(_('[inst] '), noln=True) + else: + ctx.ui.info(_('Installed package:')) + self.print_pkginfo(metadata, files,pisi.itembyrepodb.installed) + + if ctx.packagedb.has_package(arg): + metadata, files, repo = pisi.api.info_name(arg, False) + if self.options.short: + ctx.ui.info(_('[repo] '), noln=True) + else: + ctx.ui.info(_('Package found in %s repository:') % repo) + self.print_pkginfo(metadata, files, pisi.itembyrepodb.repos) + + if not ctx.packagedb.has_package(arg): + ctx.ui.info(_("%s is not found in repositories") % arg) + + if not ctx.installdb.is_installed(arg): + ctx.ui.info(_("%s is not installed") % arg) + + def print_pkginfo(self, metadata, files, repo = None): + if ctx.get_option('short'): + pkg = metadata.package + ctx.ui.info('%15s - %s' % (pkg.name, unicode(pkg.summary))) + else: + ctx.ui.info(unicode(metadata.package)) + if repo: + revdeps = [x[0] for x in + ctx.packagedb.get_rev_deps(metadata.package.name, repo)] + print _('Reverse Dependencies:'), util.strlist(revdeps) + if self.options.files or self.options.files_path: + if files: + print _('\nFiles:') + files.list.sort(key = lambda x:x.path) + for fileinfo in files.list: + if self.options.files: + print fileinfo + else: + print "/" + fileinfo.path + else: + ctx.ui.warning(_('File information not available')) + if not self.options.short: + print + + +class Check(Command): + """Verify installation + +Usage: check [ ... ] + +: package name + +A cryptographic checksum is stored for each installed +file. Check command uses the checksums to verify a package. +Just give the names of packages. + +If no packages are given, checks all installed packages. +""" + __metaclass__ = autocommand + + def __init__(self, args): + super(Check, self).__init__(args) + + name = ("check", None) + + def options(self): + group = OptionGroup(self.parser, _("check options")) + group.add_option("-c", "--component", action="store", + default=None, help=_("Check installed packages under given component")) + self.parser.add_option_group(group) + + def run(self): + self.init(database = True, write = False) + + component = ctx.get_option('component') + if component: + #FIXME: pisi api is insufficient to do this + from sets import Set as set + installed = ctx.installdb.list_installed() + component_pkgs = ctx.componentdb.get_union_packages(component, walk=True) + pkgs = list(set(installed) & set(component_pkgs)) + elif self.args: + pkgs = self.args + else: + ctx.ui.info(_('Checking all installed packages')) + pkgs = ctx.installdb.list_installed() + + for pkg in pkgs: + ctx.ui.info(_('* Checking %s... ') % pkg, noln=True) + if ctx.installdb.is_installed(pkg): + corrupt = pisi.api.check(pkg) + if corrupt: + ctx.ui.info(_('\nPackage %s is corrupt.') % pkg) + else: + ctx.ui.info(_("OK"), verbose=False) + else: + ctx.ui.info(_('Package %s not installed') % pkg) + self.finalize() + + +class Index(Command): + """Index PiSi files in a given directory + +Usage: index ... + +This command searches for all PiSi files in a directory, collects PiSi +tags from them and accumulates the information in an output XML file, +named by default 'pisi-index.xml'. In particular, it indexes both +source and binary packages. + +If you give multiple directories, the command still works, but puts +everything in a single index file. +""" + __metaclass__ = autocommand + + def __init__(self, args): + super(Index, self).__init__(args) + + name = ("index", "ix") + + def options(self): + + group = OptionGroup(self.parser, _("index options")) + + group.add_option("-a", "--absolute-urls", action="store_true", + default=False, + help=_("Store absolute links for indexed files.")) + group.add_option("-o", "--output", action="store", + default='pisi-index.xml', + help=_("Index output file")) + group.add_option("--skip-sources", action="store_true", + default=False, + help=_("Do not index PiSi spec files.")) + group.add_option("--skip-signing", action="store_true", + default=False, + help=_("Do not sign index.")) + + self.parser.add_option_group(group) + + def run(self): + + self.init(database = True, write = False) + from pisi.api import index + if len(self.args)>0: + index(self.args, ctx.get_option('output'), + skip_sources = ctx.get_option('skip_sources'), + skip_signing = ctx.get_option('skip_signing')) + elif len(self.args)==0: + ctx.ui.info(_('Indexing current directory.')) + index(['.'], ctx.get_option('output'), + skip_sources = ctx.get_option('skip_sources'), + skip_signing = ctx.get_option('skip_signing')) + self.finalize() + + +class ListInstalled(Command): + """Print the list of all installed packages + +Usage: list-installed +""" + + __metaclass__ = autocommand + + def __init__(self, args): + super(ListInstalled, self).__init__(args) + + name = ("list-installed", "li") + + def options(self): + + group = OptionGroup(self.parser, _("list-installed options")) + + group.add_option("-l", "--long", action="store_true", + default=False, help=_("Show in long format")) + group.add_option("-c", "--component", action="store", + default=None, help=_("List installed packages under given component")) + group.add_option("-i", "--install-info", action="store_true", + default=False, help=_("Show detailed install info")) + + self.parser.add_option_group(group) + + def run(self): + self.init(database = True, write = False) + installed = ctx.installdb.list_installed() + + component = ctx.get_option('component') + if component: + #FIXME: pisi api is insufficient to do this + from sets import Set as set + component_pkgs = ctx.componentdb.get_union_packages(component, walk=True) + installed = list(set(installed) & set(component_pkgs)) + + installed.sort() + if self.options.install_info: + ctx.ui.info(_('Package Name |St| Version| Rel.| Build| Distro| Date')) + print '========================================================================' + for pkg in installed: + package = ctx.packagedb.get_package(pkg, pisi.itembyrepodb.installed) + inst_info = ctx.installdb.get_info(pkg) + if self.options.long: + ctx.ui.info(unicode(package)) + ctx.ui.info(unicode(inst_info)) + elif self.options.install_info: + ctx.ui.info('%-15s |%s' % (package.name, inst_info.one_liner())) + else: + ctx.ui.info('%15s - %s' % (package.name, unicode(package.summary))) + self.finalize() + +class RebuildDb(Command): + """Rebuild Databases + +Usage: rebuilddb [ ... ] + +Rebuilds the PiSi databases + +If package specs are given, they should be the names of package +dirs under /var/lib/pisi +""" + __metaclass__ = autocommand + + def __init__(self, args): + super(RebuildDb, self).__init__(args) + + name = ("rebuild-db", "rdb") + + def options(self): + + group = OptionGroup(self.parser, _("rebuild-db options")) + + group.add_option("-f", "--files", action="store_true", + default=False, help=_("Rebuild files database")) + + self.parser.add_option_group(group) + + def run(self): + if self.args: + self.init(database=True) + for package_fn in self.args: + pisi.api.resurrect_package(package_fn, ctx.get_option('files`')) + else: + self.init(database=False) + if ctx.ui.confirm(_('Rebuild PiSi databases?')): + pisi.api.rebuild_db(ctx.get_option('files')) + + self.finalize() + + +class UpdateRepo(Command): + """Update repository databases + +Usage: update-repo [ ... ] + +: repository name + +Synchronizes the PiSi databases with the current repository. +If no repository is given, all repositories are updated. +""" + __metaclass__ = autocommand + + def __init__(self,args): + super(UpdateRepo, self).__init__(args) + + name = ("update-repo", "ur") + + def options(self): + + group = OptionGroup(self.parser, _("update-repo options")) + + group.add_option("-f", "--force", action="store_true", + default=False, + help=_("Update database in any case")) + + self.parser.add_option_group(group) + + def run(self): + self.init(database = True) + + if self.args: + repos = self.args + else: + repos = ctx.repodb.list() + + for repo in repos: + pisi.api.update_repo(repo, ctx.get_option('force')) + self.finalize() + + +class AddRepo(Command): + """Add a repository + +Usage: add-repo + +: name of repository to add +: URI of index file + +If no repo is given, add-repo pardus-devel repo is added by default + +NB: We support only local files (e.g., /a/b/c) and http:// URIs at the moment +""" + __metaclass__ = autocommand + + def __init__(self, args): + super(AddRepo, self).__init__(args) + + name = ("add-repo", "ar") + + def options(self): + + group = OptionGroup(self.parser, _("add-repo options")) + group.add_option("--at", action="store", + type="int", default=None, + help=_("Add repository at given position (0 is first)")) + self.parser.add_option_group(group) + + def run(self): + + if len(self.args)==2 or len(self.args)==0: + self.init() + if len(self.args)==2: + name = self.args[0] + indexuri = self.args[1] + else: + name = 'pardus-2007' + indexuri = 'http://paketler.pardus.org.tr/pardus-2007/pisi-index.xml.bz2' + pisi.api.add_repo(name, indexuri, ctx.get_option('at')) + if ctx.ui.confirm(_('Update PiSi database for repository %s?') % name): + try: + pisi.api.update_repo(name) + except pisi.fetcher.FetchError: + ctx.ui.warning(_("%s repository could not be reached. Removing %s from system.") % (name, name)) + pisi.api.remove_repo(name) + self.finalize() + else: + self.help() + return + + +class RemoveRepo(Command): + """Remove repositories + +Usage: remove-repo ... + +Remove all repository information from the system. +""" + __metaclass__ = autocommand + + def __init__(self,args): + super(RemoveRepo, self).__init__(args) + + name = ("remove-repo", "rr") + + def run(self): + + if len(self.args)>=1: + self.init() + for repo in self.args: + pisi.api.remove_repo(repo) + self.finalize() + else: + self.help() + return + + +class ListRepo(Command): + """List repositories + +Usage: list-repo + +Lists currently tracked repositories. +""" + __metaclass__ = autocommand + + def __init__(self, args): + super(ListRepo, self).__init__(args) + + name = ("list-repo", "lr") + + def run(self): + + self.init(database = True, write = False) + for repo in ctx.repodb.list(): + ctx.ui.info(repo) + print ' ', ctx.repodb.get_repo(repo).indexuri.get_uri() + self.finalize() + + +class ListAvailable(Command): + """List available packages in the repositories + +Usage: list-available [ ... repon ] + +Gives a brief list of PiSi packages published in the specified +repositories. If no repository is specified, we list packages in +all repositories. +""" + __metaclass__ = autocommand + + def __init__(self, args): + super(ListAvailable, self).__init__(args) + + name = ("list-available", "la") + + def options(self): + + group = OptionGroup(self.parser, _("list-available options")) + group.add_option("-l", "--long", action="store_true", + default=False, help=_("Show in long format")) + group.add_option("-c", "--component", action="store", + default=None, help=_("List available packages under given component")) + group.add_option("-U", "--uninstalled", action="store_true", + default=False, help=_("Show uninstalled packages only")) + self.parser.add_option_group(group) + + def run(self): + + self.init(database = True, write = False) + + if not (ctx.get_option('no_color') or ctx.config.get_option('uninstalled')): + ctx.ui.info(util.colorize(_('Installed packages are shown in this color'), 'green')) + + if self.args: + for arg in self.args: + self.print_packages(arg) + else: + # print for all repos + for repo in ctx.repodb.list(): + ctx.ui.info(_("Repository : %s\n") % repo) + self.print_packages(repo) + self.finalize() + + def print_packages(self, repo): + + component = ctx.get_option('component') + if component: + l = ctx.componentdb.get_packages(component, walk=True, repo=repo) + else: + l = ctx.packagedb.list_packages(repo) + installed_list = ctx.installdb.list_installed() + l.sort() + for p in l: + package = ctx.packagedb.get_package(p) + if self.options.long: + ctx.ui.info(unicode(package)) + else: + lenp = len(p) + if p in installed_list: + if ctx.config.get_option('uninstalled'): + continue + p = util.colorize(p, 'green') + p = p + ' ' * max(0, 15 - lenp) + ctx.ui.info('%s - %s ' % (p, unicode(package.summary))) + +class ListComponents(Command): + """List available components + +Usage: list-components + +Gives a brief list of PiSi components published in the +repositories. +""" + __metaclass__ = autocommand + + def __init__(self, args): + super(ListComponents, self).__init__(args) + + name = ("list-components", "lc") + + def options(self): + group = OptionGroup(self.parser, _("list-components options")) + group.add_option("-l", "--long", action="store_true", + default=False, help=_("Show in long format")) + self.parser.add_option_group(group) + + def run(self): + + self.init(database = True, write = False) + + l = ctx.componentdb.list_components() + l.sort() + for p in l: + component = ctx.componentdb.get_component(p) + if self.options.long: + ctx.ui.info(unicode(component)) + else: + lenp = len(p) + #if p in installed_list: + # p = util.colorize(p, 'cyan') + p = p + ' ' * max(0, 15 - lenp) + ctx.ui.info('%s - %s ' % (component.name, unicode(component.summary))) + self.finalize() + + +class ListSources(Command): + """List available sources + +Usage: list-sources + +Gives a brief list of sources published in the repositories. +""" + __metaclass__ = autocommand + + def __init__(self, args): + super(ListSources, self).__init__(args) + + name = ("list-sources", "ls") + + def options(self): + group = OptionGroup(self.parser, _("list-sources options")) + group.add_option("-l", "--long", action="store_true", + default=False, help=_("Show in long format")) + self.parser.add_option_group(group) + + def run(self): + + self.init(database = True, write = False) + + l = ctx.sourcedb.list() + l.sort() + for p in l: + sf, repo = ctx.sourcedb.get_spec_repo(p) + if self.options.long: + ctx.ui.info('[Repository: ' + repo + ']') + ctx.ui.info(unicode(sf.source)) + else: + lenp = len(p) + #if p in installed_list: + # p = util.colorize(p, 'cyan') + p = p + ' ' * max(0, 15 - lenp) + ctx.ui.info('%s - %s' % (sf.source.name, unicode(sf.source.summary))) + self.finalize() + +class ListUpgrades(Command): + """List packages to be upgraded + +Usage: list-upgrades + +Lists the packages that will be upgraded. +""" + __metaclass__ = autocommand + + def __init__(self, args): + super(ListUpgrades, self).__init__(args) + + name = ("list-upgrades", "lu") + + def options(self): + group = OptionGroup(self.parser, _("list-upgrades options")) + buildno_opts(self, group) + group.add_option("-l", "--long", action="store_true", + default=False, help=_("Show in long format")) + group.add_option("-c", "--component", action="store", + default=None, help=_("List upgradable packages under given component")) + group.add_option("-i", "--install-info", action="store_true", + default=False, help=_("Show detailed install info")) + self.parser.add_option_group(group) + + def run(self): + self.init(database = True, write = False) + upgradable_pkgs = pisi.api.list_upgradable() + + component = ctx.get_option('component') + if component: + #FIXME: PiSi api is insufficient to do this + from sets import Set as set + component_pkgs = ctx.componentdb.get_union_packages(component, walk=True) + upgradable_pkgs = list(set(upgradable_pkgs) & set(component_pkgs)) + + if not upgradable_pkgs: + ctx.ui.info(_('No packages to upgrade.')) + + upgradable_pkgs.sort() + if self.options.install_info: + ctx.ui.info(_('Package Name |St| Version| Rel.| Build| Distro| Date')) + print '========================================================================' + for pkg in upgradable_pkgs: + package = ctx.packagedb.get_package(pkg, pisi.itembyrepodb.installed) + inst_info = ctx.installdb.get_info(pkg) + if self.options.long: + ctx.ui.info(package) + print inst_info + elif self.options.install_info: + ctx.ui.info('%-15s | %s ' % (package.name, inst_info.one_liner())) + else: + ctx.ui.info('%15s - %s ' % (package.name, package.summary)) + self.finalize() + + +class ListPending(Command): + """List pending packages + +Lists packages waiting to be configured. +""" + + __metaclass__ = autocommand + + def __init__(self, args): + super(ListPending, self).__init__(args) + + name = ("list-pending", "lp") + + def run(self): + self.init(database = True, write = False) + + A = ctx.installdb.list_pending() + order = pisi.api.generate_pending_order(A) + + if len(order): + for p in order: + print p + else: + ctx.ui.info(_('There are no packages waiting to be configured')) + self.finalize() + + +class Search(Info): + """Search packages + +Usage: search ... + +Finds a package containing specified search terms +in summary, description, and package name fields. +""" + __metaclass__ = autocommand + + def __init__(self, args): + super(Search, self).__init__(args) + + name = ("search", "sr") + + def options(self): + group = OptionGroup(self.parser, _("search options")) + super(Search, self).add_options(group) + group.remove_option("--component") + group.remove_option("--short") + group.remove_option("--xml") + group.add_option("-l", "--long", action="store_true", + default=False, help=_("Show details")) + self.parser.add_option_group(group) + + def run(self): + + self.init(database = True, write = False) + + if not self.args: + self.help() + return + + r = pisi.api.search_package_terms(self.args) + ctx.ui.info(_('%s packages found') % len(r)) + + ctx.config.options.short = not ctx.config.options.long + for pkg in r: + self.info_package(pkg) + + self.finalize() + +class SearchFile(Command): + """Search for a file + +Usage: search-file ... + +Finds the installed package which contains the specified file. +""" + __metaclass__ = autocommand + + def __init__(self, args): + super(SearchFile, self).__init__(args) + + name = ("search-file", "sf") + + def options(self): + group = OptionGroup(self.parser, _("search-file options")) + group.add_option("-l", "--long", action="store_true", + default=False, help=_("Show in long format")) + group.add_option("-f", "--fuzzy", action="store_true", + default=False, help=_("Fuzzy search")) + group.add_option("-q", "--quiet", action="store_true", + default=False, help=_("Show only package name")) + self.parser.add_option_group(group) + + # what does exact mean? -- exa + @staticmethod + def search_exact(path): + files = [] + path = path.lstrip('/') #FIXME: this shouldn't be necessary :/ + + if not ctx.config.options.fuzzy: + if ctx.filesdb.has_file(path): + files.append(ctx.filesdb.get_file(path)) + else: + #FIXME: this linear search thing is not working well -- exa + files = ctx.filesdb.match_files(path) + + if files: + for (pkg_name, file_info) in files: + if ctx.config.options.quiet: + ctx.ui.info(pkg_name) + else: + ctx.ui.info(_("Package %s has file %s") % (pkg_name, file_info.path)) + if ctx.config.options.long: + ctx.ui.info(_('Type: %s, Hash: %s') % (file_info.type, + file_info.hash)) + else: + ctx.ui.error(_("Path '%s' does not belong to an installed package") % path) + + def run(self): + + self.init(database = True, write = False) + + if not self.args: + self.help() + return + + # search among existing files + for path in self.args: + if not ctx.config.options.quiet: + ctx.ui.info(_('Searching for %s') % path) + import os.path + if os.path.exists(path): + path = os.path.realpath(path) + self.search_exact(path) + + self.finalize() + +# texts + +usage_text1 = _("""%prog [options] [arguments] + +where is one of: + +""") + +usage_text2 = _(""" +Use \"%prog help \" for help on a specific command. +""") + +usage_text = (usage_text1 + Command.commands_string() + usage_text2) diff --git a/branches/pisi-1.1/pisi/cli/pisicli.py b/branches/pisi-1.1/pisi/cli/pisicli.py new file mode 100644 index 00000000..a47fe8d8 --- /dev/null +++ b/branches/pisi-1.1/pisi/cli/pisicli.py @@ -0,0 +1,108 @@ +# -*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. +# + +import sys +from optparse import OptionParser + +import gettext +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +import pisi +import pisi.cli +from pisi.cli import printu +from pisi.uri import URI +from pisi.cli.commands import * + +class ParserError(pisi.Exception): + pass + +class PreParser(OptionParser): + """consumes any options, and finds arguments from command line""" + + def __init__(self, version): + OptionParser.__init__(self, usage=usage_text, version=version) + + def error(self, msg): + raise ParserError, msg + + def parse_args(self, args=None): + self.opts = [] + self.rargs = self._get_args(args) + self._process_args() + return (self.opts, self.args) + + def _process_args(self): + args = [] + rargs = self.rargs + if not self.allow_interspersed_args: + first_arg = False + while rargs: + arg = rargs[0] + def option(): + if not self.allow_interspersed_args and first_arg: + self.error(_('Options must precede non-option arguments')) + arg = rargs[0] + if arg.startswith('--'): + self.opts.append(arg[2:]) + else: + self.opts.append(arg[1:]) + del rargs[0] + return + # We handle bare "--" explicitly, and bare "-" is handled by the + # standard arg handler since the short arg case ensures that the + # len of the opt string is greater than 1. + if arg == "--": + del rargs[0] + break + elif arg[0:2] == "--": + # process a single long option (possibly with value(s)) + option() + elif arg[:1] == "-" and len(arg) > 1: + # process a cluster of short options (possibly with + # value(s) for the last one only) + option() + else: # then it must be an argument + args.append(arg) + del rargs[0] + self.args = args + + +class PisiCLI(object): + + def __init__(self, orig_args=None): + # first construct a parser for common options + # this is really dummy + self.parser = PreParser(version="%prog " + pisi.__version__) + try: + opts, args = self.parser.parse_args(args=orig_args) + if len(args)==0: # more explicit than using IndexError + if 'version' in opts: + self.parser.print_version() + sys.exit(0) + elif 'help' in opts or 'h' in opts: + self.die() + raise Error(_('No command given')) + cmd_name = args[0] + except ParserError: + raise Error(_('Command line parsing error')) + + self.command = Command.get_command(cmd_name, args=orig_args) + if not self.command: + raise Error(_("Unrecognized command: %s") % cmd_name) + + def die(self): + printu('\n' + self.parser.format_help()) + sys.exit(1) + + def run_command(self): + self.command.run() diff --git a/branches/pisi-1.1/pisi/comariface.py b/branches/pisi-1.1/pisi/comariface.py new file mode 100644 index 00000000..4e97508a --- /dev/null +++ b/branches/pisi-1.1/pisi/comariface.py @@ -0,0 +1,135 @@ +# -*- coding: utf-8 -*- +# +# Copyright (C) 2005-2006, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. +# + +import os +import time +import select + +import gettext +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +import pisi +import pisi.context as ctx + +class Error(pisi.Error): + pass + +try: + import comar +except ImportError: + raise Error(_("comar package is not fully installed")) + +def get_comar(): + """Connect to the comar daemon and return the handle""" + + sockname = "/var/run/comar.socket" + # YALI starts comar chrooted in the install target, but uses PiSi outside of + # the chroot environment, so PiSi needs to use a different socket path to be + # able to connect true comar (usually /mnt/target/var/run/comar.socket). + if ctx.comar_sockname: + sockname = ctx.comar_sockname + + # This function is sometimes called when comar has recently started + # or restarting after an update. So we give comar a chance to become + # active in a reasonable time. + timeout = 7 + while timeout > 0: + try: + com = comar.Link(sockname) + return com + except comar.CannotConnect: + pass + time.sleep(0.2) + timeout -= 0.2 + raise Error(_("cannot connect to comar")) + +def wait_for_result(com, package_name=None): + multiple = False + while True: + try: + reply = com.read_cmd() + except select.error: + if ctx.keyboard_interrupt_pending(): + return + raise + except comar.LinkClosed: + # Comar postInstall does a "service comar restart" which cuts + # our precious communication link, so we waitsss + if package_name == "comar": + try: + get_comar() + except Error: + raise Error, _("Could not restart comar") + return + else: + if ctx.keyboard_interrupt_pending(): + return + raise Error, _("connection with comar unexpectedly closed") + + cmd = reply[0] + if cmd == com.RESULT and not multiple: + return + elif cmd == com.NONE and not multiple: + # no post/pre function, that is ok + return + elif cmd == com.RESULT_START: + multiple = True + elif cmd == com.RESULT_END: + return + elif cmd == com.FAIL: + raise Error, _("Configuration error: %s") % reply[2] + elif cmd == com.ERROR: + raise Error, _("Script error: %s") % reply[2] + elif cmd == com.DENIED: + raise Error, _("comar denied our access") + +def post_install(package_name, provided_scripts, scriptpath, metapath, filepath): + """Do package's post install operations""" + + ctx.ui.info(_("Configuring %s package") % package_name) + self_post = False + com = get_comar() + + for script in provided_scripts: + ctx.ui.debug(_("Registering %s comar script") % script.om) + if script.om == "System.Package": + self_post = True + com.register(script.om, package_name, os.path.join(scriptpath, script.script)) + wait_for_result(com) + + ctx.ui.debug(_("Calling post install handlers")) + com.call("System.PackageHandler.setupPackage", [ "metapath", metapath, "filepath", filepath ]) + wait_for_result(com) + + if self_post: + ctx.ui.debug(_("Running package's post install script")) + com.call_package("System.Package.postInstall", package_name) + wait_for_result(com, package_name) + +def pre_remove(package_name, metapath, filepath): + """Do package's pre removal operations""" + + ctx.ui.info(_("Configuring %s package for removal") % package_name) + com = get_comar() + + ctx.ui.debug(_("Running package's pre remove script")) + com.call_package("System.Package.preRemove", package_name) + wait_for_result(com) + + ctx.ui.debug(_("Calling pre remove handlers")) + com.call("System.PackageHandler.cleanupPackage", [ "metapath", metapath, "filepath", filepath ]) + wait_for_result(com) + + ctx.ui.debug(_("Unregistering comar scripts")) + com.remove(package_name) + wait_for_result(com) diff --git a/branches/pisi-1.1/pisi/component.py b/branches/pisi-1.1/pisi/component.py new file mode 100644 index 00000000..61b44a18 --- /dev/null +++ b/branches/pisi-1.1/pisi/component.py @@ -0,0 +1,251 @@ +# -*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. +# + +import gettext +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +import pisi +import pisi.context as ctx +import pisi.pxml.xmlfile as xmlfile +import pisi.pxml.autoxml as autoxml +import pisi.lockeddbshelve as shelve +import pisi.itembyrepodb + +class Error(pisi.Error): + pass + +__metaclass__ = autoxml.autoxml + +class Obsolete: + + __metaclass__ = autoxml.autoxml + + s_Package = [autoxml.String, autoxml.mandatory] + + def __str__(self): + return self.package + +class Distribution(xmlfile.XmlFile): + + __metaclass__ = autoxml.autoxml + + tag = "PISI" + + t_SourceName = [autoxml.Text, autoxml.mandatory] # name of distribution (source) + t_Description = [autoxml.LocalText, autoxml.mandatory] + t_Version = [autoxml.Text, autoxml.optional] + t_Type = [autoxml.Text, autoxml.mandatory] + t_Dependencies = [ [autoxml.Text], autoxml.optional, "Dependencies/Distribution"] + + t_BinaryName = [autoxml.Text, autoxml.optional] # name of repository (binary distro) + t_Architecture = [autoxml.Text, autoxml.optional] # architecture identifier + + t_Obsoletes = [ [Obsolete], autoxml.optional, "Obsoletes/Package"] + +class Component(xmlfile.XmlFile): + "representation for component declarations" + + __metaclass__ = autoxml.autoxml + + tag = "PISI" + + t_Name = [autoxml.String, autoxml.mandatory] # fully qualified name + + # component name in other languages, for instance in Turkish + # LocalName for system.base could be sistem.taban or "Taban Sistem", + # this could be useful for GUIs + + t_LocalName = [autoxml.LocalText, autoxml.mandatory] + + # Information about the component + t_Summary = [autoxml.LocalText, autoxml.mandatory] + t_Description = [autoxml.LocalText, autoxml.mandatory] + t_Icon = [ autoxml.String, autoxml.optional] + t_VisibleTo = [autoxml.String, autoxml.optional] + + # Dependencies to other components + t_Dependencies = [ [autoxml.String], autoxml.optional, "Dependencies/Component"] + + # the parts of this component. + # to be filled by the component database, thus it is optional. + t_Packages = [ [autoxml.String], autoxml.optional, "Parts/Package"] + + t_Sources = [ [autoxml.String], autoxml.optional, "Parts/Source"] + +class ComponentDB(object): + """a database of components""" + + def __init__(self): + self.d = pisi.itembyrepodb.ItemByRepoDB('component') + + def close(self): + self.d.close() + + def destroy(self): + self.d.destroy() + + def has_component(self, name, repo = pisi.itembyrepodb.repos, txn = None): + name = str(name) + return self.d.has_key(name, repo, txn) + + def get_component(self, name, repo=None, txn = None): + try: + return self.d.get_item(name, repo, txn=txn) + except pisi.itembyrepodb.NotfoundError, e: + raise Error(_('Component %s not found') % name) + + def get_component_repo(self, name, repo=None, txn = None): + try: + return self.d.get_item_repo(name, repo, txn=txn) + except pisi.itembyrepodb.NotfoundError, e: + raise Error(_('Component %s not found') % name) + + def get_union_comp(self, name, txn = None, repo = pisi.itembyrepodb.repos ): + """get a union of all repository components packages, not just the first repo in order. + get only basic repo info from the first repo""" + def proc(txn): + s = self.d.d.get(name, txn=txn) + pkgs = set() + srcs = set() + for repostr in self.d.order(repo = repo): + if s.has_key(repostr): + pkgs |= set(s[repostr].packages) + srcs |= set(s[repostr].sources) + comp = self.get_component(name) + comp.packages = list(pkgs) + comp.sources = list(srcs) + return comp + return self.d.txn_proc(proc, txn) + + def list_components(self, repo=None): + return self.d.list(repo) + + # walk: walks through the underlying components' packages + def get_union_packages(self, component_name, walk=False, repo=pisi.itembyrepodb.repos, txn = None): + """returns union of all repository component's packages, not just the first repo's + component's in order""" + + component = self.get_union_comp(component_name, txn, repo) + if not walk: + return component.packages + + packages = [] + packages.extend(component.packages) + for dep in component.dependencies: + packages.extend(self.get_union_packages(dep, walk, repo, txn)) + + return packages + + # walk: walks through the underlying components' packages + def get_packages(self, component_name, walk=False, repo=None, txn = None): + """returns the given component's and underlying recursive components' packages""" + + component = self.get_component(component_name, repo, txn) + if not walk: + return component.packages + + packages = [] + packages.extend(component.packages) + for dep in component.dependencies: + packages.extend(self.get_packages(dep, walk, repo, txn)) + + return packages + + def add_child(self, component, repo, txn = None): + """update component tree""" + parent_name = ".".join(component.name.split(".")[:-1]) + if not parent_name: # root component + return + + if self.has_component(parent_name, repo, txn): + parent = self.get_component(parent_name, repo, txn) + else: + parent = Component(name = parent_name) + + if component.name not in parent.dependencies: + parent.dependencies.append(component.name) + self.d.add_item(parent_name, parent, repo, txn) + + def update_component(self, component, repo, txn = None): + def proc(txn): + if self.has_component(component.name, repo, txn): + # preserve list of sources, packages and dependencies + current = self.d.get_item(component.name, repo, txn) + component.packages = current.packages + component.sources = current.sources + component.dependencies = current.dependencies + self.d.add_item(component.name, component, repo, txn) + self.add_child(component, repo, txn) + self.d.txn_proc(proc, txn) + + def add_package(self, component_name, package, repo, txn = None): + def proc(txn): + assert component_name + if self.has_component(component_name, repo, txn): + component = self.get_component(component_name, repo, txn) + else: + component = Component( name = component_name ) + if not package in component.packages: + component.packages.append(package) + self.d.add_item(component_name, component, repo, txn) # update + self.add_child(component, repo, txn) + self.d.txn_proc(proc, txn) + + def remove_package(self, component_name, package, repo = None, txn = None): + def proc(txn, repo): + if not self.has_component(component_name, repo, txn): + raise Error(_('Information for component %s not available') % component_name) + if not repo: + repo = self.d.which_repo(component_name, txn=txn) # get default repo then + component = self.get_component(component_name, repo, txn) + if package in component.packages: + component.packages.remove(package) + self.d.add_item(component_name, component, repo, txn) # update + + ctx.txn_proc(lambda x: proc(txn, repo), txn) + + def add_spec(self, component_name, spec, repo, txn = None): + def proc(txn): + assert component_name + if self.has_component(component_name, repo, txn): + component = self.get_component(component_name, repo, txn) + else: + component = Component( name = component_name ) + if not spec in component.sources: + component.sources.append(spec) + self.d.add_item(component_name, component, repo, txn) # update + self.add_child(component, repo, txn) + self.d.txn_proc(proc, txn) + + def remove_spec(self, component_name, spec, repo = None, txn = None): + def proc(txn, repo): + if not self.has_component(component_name, repo, txn): + raise Error(_('Information for component %s not available') % component_name) + if not repo: + repo = self.d.which_repo(component_name, txn=txn) # get default repo then + component = self.get_component(component_name, repo, txn) + if spec in component.sources: + component.sources.remove(spec) + self.d.add_item(component_name, component, repo, txn) # update + + ctx.txn_proc(lambda x: proc(txn, repo), txn) + + def clear(self, txn = None): + self.d.clear(txn) + + def remove_component(self, name, repo = None, txn = None): + name = str(name) + self.d.remove_item(name, repo, txn) + + def remove_repo(self, repo, txn = None): + self.d.remove_repo(repo, txn=txn) diff --git a/branches/pisi-1.1/pisi/config.py b/branches/pisi-1.1/pisi/config.py new file mode 100644 index 00000000..e21a53d7 --- /dev/null +++ b/branches/pisi-1.1/pisi/config.py @@ -0,0 +1,123 @@ +# -*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. +# + +""" +PiSi Configuration module is used for gathering and providing +regular PiSi configurations. +""" + +import os +import copy + +import gettext +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +import pisi +import pisi.context as ctx +import pisi.configfile +import pisi.util + +class Error(pisi.Error): + pass + +class Options(object): + def __getattr__(self, name): + if not self.__dict__.has_key(name): + return None + else: + return self.__dict__[name] + + def __setattr__(self, name, value): + self.__dict__[name] = value + +class Config(object): + """Config Singleton""" + + def __init__(self, options = Options()): + self.options = options + self.values = pisi.configfile.ConfigurationFile("/etc/pisi/pisi.conf") + + destdir = self.get_option('destdir') + if destdir: + if destdir.strip().startswith('/'): + self.destdir = destdir + else: + self.destdir = pisi.util.join_path(os.getcwd(), destdir) + else: + self.destdir = self.values.general.destinationdirectory + + if not os.path.exists(self.destdir): + ctx.ui.warning( _('Destination directory %s does not exist. Creating it.') % self.destdir) + os.makedirs(self.destdir) + + # get the initial environment variables. this is needed for + # build process. + self.environ = copy.deepcopy(os.environ) + + def set_option(self, opt, val): + setattr(self.options, opt, val) + + def get_option(self, opt): + if self.options: + if hasattr(self.options, opt): + return getattr(self.options, opt) + return None + + # directory accessor functions + # here is how it goes + # x_dir: system wide directory for storing info type x + # pkg_x_dir: per package directory for storing info type x + + def dest_dir(self): + return self.destdir + + def subdir(self, path): + subdir = pisi.util.join_path(self.dest_dir(), path) + pisi.util.check_dir(subdir) + return subdir + + def log_dir(self): + return self.subdir(self.values.dirs.log_dir) + + def lib_dir(self): + return self.subdir(self.values.dirs.lib_dir) + + def db_dir(self): + return self.subdir(self.values.dirs.db_dir) + + def archives_dir(self): + return self.subdir(self.values.dirs.archives_dir) + + def packages_dir(self): + return self.subdir(self.values.dirs.packages_dir) + + def compiled_packages_dir(self): + return self.subdir(self.values.dirs.compiled_packages_dir) + + def index_dir(self): + return self.subdir(self.values.dirs.index_dir) + + def tmp_dir(self): + sysdir = self.subdir(self.values.dirs.tmp_dir) + if os.environ.has_key('USER'): + userdir = self.subdir('/tmp/pisi-' + os.environ['USER']) + else: + userdir = self.subdir('/tmp/pisi-root') + # check write access + if os.access(sysdir, os.W_OK): + return sysdir + else: + return userdir + +#TODO: remove this +config = Config() diff --git a/branches/pisi-1.1/pisi/configfile.py b/branches/pisi-1.1/pisi/configfile.py new file mode 100644 index 00000000..1e286270 --- /dev/null +++ b/branches/pisi-1.1/pisi/configfile.py @@ -0,0 +1,304 @@ +# -*- coding: utf-8 -*- +# +# Copyright (C) 2005 - 2007, TUBITAK/UEKAE +# +# This program is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# Please read the COPYING file. +# + +# PiSi Configuration File module, obviously, is used to read from the +# configuration file. Module also defines default values for +# configuration parameters. +# +# Configuration file is located in /etc/pisi/pisi.conf by default, +# having an INI like format like below. +# +#[general] +#destinationdirectory = / +#autoclean = False +# +#[build] +#host = i686-pc-linux-gnu +#generateDebug = False +#enableSandbox = True +#jobs = "-j1" +#CFLAGS= -mtune=i686 -march=i686 -O2 -pipe -fomit-frame-pointer +#CXXFLAGS= -mtune=i686 -march=i686 -O2 -pipe -fomit-frame-pointer +#LDFLAGS= +#buildno=True # necessary for generating build nos +#buildhelper = None / ccache / icecream +#compressionlevel = 7 +#fallback = "ftp://ftp.pardus.org.tr/pub/pisi/source" +# +#[directories] +#lib_dir = /var/lib/pisi +#db_dir = /var/db/pisi +#archives_dir = /var/cache/pisi/archives +#packages_dir = /var/cache/pisi/packages +#compiled_packages_dir = "/var/cache/pisi/packages" +#index_dir = /var/cache/pisi/index +#tmp_dir = /var/pisi +#kde_dir = /usr/kde/3.5 +#qt_dir = /usr/qt/3 + +import os +import re +import StringIO +import ConfigParser + +import gettext +__trans = gettext.translation('pisi', fallback=True) +_ = __trans.ugettext + +import pisi + +class Error(pisi.Error): + pass + +class GeneralDefaults: + """Default values for [general] section""" + destinationdirectory = "/" + autoclean = False + distribution = "Pardus" + distribution_release = "2007" + http_proxy = os.getenv("HTTP_PROXY") or None + https_proxy = os.getenv("HTTPS_PROXY") or None + ftp_proxy = os.getenv("FTP_PROXY") or None + package_cache = False + package_cache_limit = 0 + +class BuildDefaults: + """Default values for [build] section""" + host = "i686-pc-linux-gnu" + jobs = "-j1" + generateDebug = False + enableSandbox = True + cflags = "-mtune=i686 -O2 -pipe -fomit-frame-pointer" + cxxflags = "-mtune=i686 -O2 -pipe -fomit-frame-pointer" + ldflags = "" + buildno = False + buildhelper = None + compressionlevel = 7 + fallback = "ftp://ftp.pardus.org.tr/pub/pisi/source" + +class DirectoriesDefaults: + "Default values for [directories] section" + lib_dir = "/var/lib/pisi" + log_dir = "/var/log" + db_dir = "/var/db/pisi" + archives_dir = "/var/cache/pisi/archives" + packages_dir = "/var/cache/pisi/packages" + compiled_packages_dir = "/var/cache/pisi/packages" + index_dir = "/var/lib/pisi/index" + tmp_dir = "/var/pisi" + kde_dir = "/usr/kde/3.5" + qt_dir = "/usr/qt/3" + +class ConfigurationSection(object): + """ConfigurationSection class defines a section in the configuration + file, using defaults (above) as a fallback.""" + def __init__(self, section, items=[]): + self.items = items + + if section == "general": + self.defaults = GeneralDefaults + elif section == "build": + self.defaults = BuildDefaults + elif section == "directories": + self.defaults = DirectoriesDefaults + else: + e = _("No section by name '%s'") % section + raise Error, e + + self.section = section + + def __getattr__(self, attr): + + # first search for attribute in the items provided in the + # configuration file. + if self.items: + for item in self.items: + if item[0] == attr: + # all values are returned as string types by ConfigParser. + # evaluate "True" or "False" strings to boolean. + if item[1] in ["True", "False", "None"]: + return eval(item[1]) + else: + return item[1] + + # then fall back to defaults + if hasattr(self.defaults, attr): + return getattr(self.defaults, attr) + + return "" + + # We'll need to access configuration keys by their names as a + # string. Like; ["default"]... + def __getitem__(self, key): + return self.__getattr__(key) + +class ConfigurationFile(object): + """Parse and get configuration values from the configuration file""" + def __init__(self, filePath): + self.parser = ConfigParser.ConfigParser() + self.filePath = filePath + + self.parser.read(self.filePath) + + try: + generalitems = self.parser.items("general") + except ConfigParser.NoSectionError: + generalitems = [] + self.general = ConfigurationSection("general", generalitems) + + try: + builditems = self.parser.items("build") + except ConfigParser.NoSectionError: + builditems = [] + self.build = ConfigurationSection("build", builditems) + + try: + dirsitems = self.parser.items("directories") + except ConfigParser.NoSectionError: + dirsitems = [] + self.dirs = ConfigurationSection("directories", dirsitems) + + # get, set and write_config methods added for manipulation of pisi.conf file from Comar to solve bug #5668. + # Current ConfigParser does not keep the comments and white spaces, which we do not want for pisi.conf. There + # are patches floating in the python sourceforge to add this feature. The write_config code is from python + # sourceforge tracker id: #1410680, modified a little to make it turn into a function. + + def get(self, section, option): + try: + return self.parser.get(section, option) + except ConfigParser.NoOptionError: + return None + + def set(self, section, option, value): + self.parser.set(section, option, value) + + def write_config(self, add_missing=True): + sections = {} + current = StringIO.StringIO() + replacement = [current] + sect = None + opt = None + written = [] + optcre = re.compile( + r'(?P