bobbuildtool / bob Goto Github PK
View Code? Open in Web Editor NEWBob build tool - Functional cross platform build-automation tool
Home Page: https://bobbuildtool.dev/
License: GNU General Public License v3.0
Bob build tool - Functional cross platform build-automation tool
Home Page: https://bobbuildtool.dev/
License: GNU General Public License v3.0
using bob with cygwin @ win7 64bit os ....
Traceback (most recent call last):
File "/cygdrive/d/projects/bob/pym/bob/scrips.py", line 79, in bob
availableCommands[verb][0](recipes, argv, bobRoot)
File "/cygdrive/d/projects/bob/pym/bob/scrips.py", line 26, in __build
doBuild(*args, **kwargs)
File "/cygdrive/d/projects/bob/pym/bob/cmds/build.py", line 761, in doBuild
commonBuildDevelop(recipes, parser, argv, bobRoot, False)
File "/cygdrive/d/projects/bob/pym/bob/cmds/build.py", line 750, in commonBuildDevelop
prettyResultPath = builder.cook([package.getPackageStep()], package)
File "/cygdrive/d/projects/bob/pym/bob/cmds/build.py", line 492, in cook
ret = self._cookPackageStep(step, done, depth)
File "/cygdrive/d/projects/bob/pym/bob/cmds/build.py", line 638, in _cookPackageStep
self.cook(packageStep.getAllDepSteps(), packageStep.getPackage(), done, depth+1)
File "/cygdrive/d/projects/bob/pym/bob/cmds/build.py", line 489, in cook
self._cookBuildStep(step, done, depth)
File "/cygdrive/d/projects/bob/pym/bob/cmds/build.py", line 568, in _cookBuildStep
self.cook(buildStep.getAllDepSteps(), buildStep.getPackage(), done, depth+1)
File "/cygdrive/d/projects/bob/pym/bob/cmds/build.py", line 486, in cook
self._cookCheckoutStep(step, done, depth)
File "/cygdrive/d/projects/bob/pym/bob/cmds/build.py", line 558, in _cookCheckoutStep
os.path.join(checkoutStep.getWorkspacePath(), "..", "cache.bin") ))
File "/cygdrive/d/projects/bob/pym/bob/utils.py", line 250, in hashDirectory
return DirHasher(index).hashDirectory(path)
File "/cygdrive/d/projects/bob/pym/bob/utils.py", line 245, in hashDirectory
return self.__hashDir(os.fsencode(path))
File "/cygdrive/d/projects/bob/pym/bob/utils.py", line 236, in __hashDir
dirList = [ self.__hashEntry(prefix, e, f, s) for (e, f, s) in entries ]
File "/cygdrive/d/projects/bob/pym/bob/utils.py", line 236, in <listcomp>
dirList = [ self.__hashEntry(prefix, e, f, s) for (e, f, s) in entries ]
File "/cygdrive/d/projects/bob/pym/bob/utils.py", line 206, in __hashEntry
digest = self.__index.check(prefix, entry, s, hashFile)
File "/cygdrive/d/projects/bob/pym/bob/utils.py", line 182, in check
self.__writeEntry(name, st, digest)
File "/cygdrive/d/projects/bob/pym/bob/utils.py", line 161, in __writeEntry
digest, len(name)))
struct.error: 'L' format requires 0 <= number <= 4294967295
some example inodes at this strange OS:
st_ino: 5154332089145070774
st_ino: -5345198597064824875
st_ino: 5004812183883991479
st_ino: 5004812183883991484
st_ino: 6047662295476103412
st_ino: 4651318196742878360
st_ino: 1496054466390112965
st_ino: -4896570410960622793
st_ino: 9132592693173250555
possible fix:
diff --git a/pym/bob/utils.py b/pym/bob/utils.py
index 0519c2c..691de5d 100644
--- a/pym/bob/utils.py
+++ b/pym/bob/utils.py
@@ -97,7 +97,7 @@ class DirHasher:
])
class FileIndex:
- CACHE_ENTRY_FMT = '=QQLLLQ20sH'
+ CACHE_ENTRY_FMT = '=QQLqLQ20sH'
CACHE_ENTRY_SIZE = struct.calcsize(CACHE_ENTRY_FMT)
class Stat:
When starting test/run-tests.sh
from inside a venv and the system lacks some dependencies, the test fails. This looks like the test is somehow not started inside the venv.
pip list | grep python-magic
is empty. Then entering the venv
source /.../pyvenv/bob/bin/activate
pip list | grep python-magic
returns the package name and version as expected. Now running test/run_tests.sh
gives this error:
Run test/generator
An internal Exception has occured. This should not have happenend.
Please open an issue at https://github.com/BobBuildTool/bob with the following backtrace:
Traceback (most recent call last):
File "/.../bob/pym/bob/scripts.py", line 98, in bob
availableCommands[verb][1](argv, bobRoot)
File "/.../bob/pym/bob/scripts.py", line 47, in __project
doProject(*args, **kwargs)
File "/.../bob/pym/bob/cmds/build.py", line 956, in doProject
from ..generators.QtCreatorGenerator import qtProjectGenerator
File "/.../bob/pym/bob/generators/QtCreatorGenerator.py", line 22, in <module>
import magic
ImportError: No module named 'magic'
--- /dev/fd/63 2016-10-05 18:38:58.416804654 +0200
+++ output-plugin.txt 2016-10-05 18:19:31.212152069 +0200
@@ -0,0 +1,3 @@
+PLUGIN: root
+PLUGIN: []
+PLUGIN: ['--download=no']
test/generator failed
Binary artifacts should have some meta data attached to them to get the recipe/package that created them. Additionally there should be some information about the input of the package, e.g.:
This information must be structured so that it can be parsed by other tools.
What I want to do: for debugging a build, I'd like to execute commands in the workspace. Could be gdb
. Could be make test
. Could be make myfile.i
. Could be ./configure
. Right now, I have to figure out the workspace directory, and do something like build.sh shell
. However, even if I do that, I'm still missing shell functions and most variables I defined in my recipes. Wouldn't it be nice to have a command like bob shell <mypackage>
?
Idea: I made a prototype of a bob shell
command (in perl, sorry; see attached file). Right now, this parses the script
that bob places in each directory using some ad-hockery, and generates a new rcfile, which it then gives to an interactive bash. This shell has environment variables set and functions defined (the script
is parsed up to the first line my ad-hoc parser does not understand, which is usually a command invocation).
How could we do this in bob directly? porting the perl to python would be trivial. A much better solution, however, would be to allow users to split setup and execution of a step. Instead of, say, buildScript, you specify buildSetup + buildScript. For building, both are concatenated to produce the same result as now. For the shell, just execute the buildSetup parts.
In e3d254c there is an additional change that adds "tags/"
to the parameter given by tag:
.
We use the tag
parameter of GitScm to get the branch, tag or commit from a multipackage with a variable:
checkoutSCM:
scm: git
url: [email protected]:example/example_common.git
tag: ${SCM_REVISION}
multiPackage:
lib-v2.3.0:
root: true
environment:
SCM_REVISION: "v2.3.0"
Is there another way for doing this? It's a workaround to handle different versions of a package inside one recipe, to tackle the problem of recipe reuse.
There is no https support for Jenkins.
Jenkins implementation is limited to unix hosts running jenkins. There are some toolchain's only available for windows... :-(
I want to update the stepEnv from the pathFmt plugin.
$ git clone https://github.com/BobBuildTool/bob-tutorials.git
$ bob-tutorials/binary-repository $ bob dev root/
>> root
CHECKOUT dev/src/root/1/workspace
>> root/lib
CHECKOUT dev/src/lib/1/workspace
>> root
>> root/lib
DOWNLOAD dev/dist/lib/1/workspace .. error
Build error: Cannot download artifact: [Errno 111] Connection refused
Failed package: root/lib
👎
steps to reproduce:
$<<file>>
The '-' is not allowed in ubuntu's bash. (GNU bash, Version 4.3.11(1)-release (x86_64-pc-linux-gnu))
Since there are several path-name problems using bob on in a cygwin bash I'm currently trying to get bob running in a mingw-bash.
>> root
>> root/lib
An internal Exception has occured. This should not have happenend.
Please open an issue at https://github.com/BobBuildTool/bob with the following backtrace:
Traceback (most recent call last):
File "C:/projects/bob/pym\bob\cmds\build.py", line 846, in commonBuildDevelop
prettyResultPath = builder.cook([package.getPackageStep()], package)
File "C:/projects/bob/pym\bob\cmds\build.py", line 549, in cook
ret = self._cookPackageStep(step, done, depth)
File "C:/projects/bob/pym\bob\cmds\build.py", line 719, in _cookPackageStep
self.cook(packageStep.getAllDepSteps(), packageStep.getPackage(), done, depth+1)
File "C:/projects/bob/pym\bob\cmds\build.py", line 546, in cook
self._cookBuildStep(step, done, depth)
File "C:/projects/bob/pym\bob\cmds\build.py", line 623, in _cookBuildStep
self.cook(buildStep.getAllDepSteps(), buildStep.getPackage(), done, depth+1)
File "C:/projects/bob/pym\bob\cmds\build.py", line 549, in cook
ret = self._cookPackageStep(step, done, depth)
File "C:/projects/bob/pym\bob\cmds\build.py", line 680, in _cookPackageStep
BobState().setDirectoryState(prettyPackagePath, packageDigest)
File "C:/projects/bob/pym\bob\state.py", line 131, in setDirectoryState
self.__save()
File "C:/projects/bob/pym\bob\state.py", line 69, in __save
os.rename(tmpFile, self.__path)
FileExistsError: [WinError 183] Eine Datei kann nicht erstellt werden, wenn sie bereits vorhanden ist: '.bob-state.pickle.new' -> '.bob-state.pickle'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/projects/bob/pym\bob\scrips.py", line 79, in bob
availableCommands[verb][0](recipes, argv, bobRoot)
File "C:/projects/bob/pym\bob\scrips.py", line 30, in __develop
doDevelop(*args, **kwargs)
File "C:/projects/bob/pym\bob\cmds\build.py", line 864, in doDevelop
commonBuildDevelop(recipes, parser, argv, bobRoot, True)
File "C:/projects/bob/pym\bob\cmds\build.py", line 849, in commonBuildDevelop
builder.saveBuildState()
File "C:/projects/bob/pym\bob\cmds\build.py", line 361, in saveBuildState
BobState().setBuildState(dict(self.__wasRun))
File "C:/projects/bob/pym\bob\state.py", line 188, in setBuildState
self.__save()
File "C:/projects/bob/pym\bob\state.py", line 69, in __save
os.rename(tmpFile, self.__path)
FileExistsError: [WinError 183] Eine Datei kann nicht erstellt werden, wenn sie bereits vorhanden ist: '.bob-state.pickle.new' -> '.bob-state.pickle'
When creating Jobs with "jenkins push" and "--no-sandbox" jobs can't be build because of missing sandbox dependencies. The jobs won't be generated but it wants to use it for calculating the buildid.
Parsing of big projects with many variants is slow and memory consumption is getting a concern too. Instead of traversing the whole dependency tree the descent should be stopped if a package has already been calculated.
An internal Exception has occured. This should not have happenend.
Please open an issue at https://github.com/BobBuildTool/bob with the following backtrace:
Traceback (most recent call last):
File "/home/user/BobBuildTool/pym/bob/scripts.py", line 114, in bob
availableCommands[verb][1](argv, bobRoot)
File "/home/user/BobBuildTool/pym/bob/scripts.py", line 52, in __project
doProject(*args, **kwargs)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 998, in doProject
doDevelop(devArgs, bobRoot)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 903, in doDevelop
commonBuildDevelop(parser, argv, bobRoot, True)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 876, in commonBuildDevelop
builder.cook([p], p.getPackage(), args.checkout_only)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 504, in cook
self._cookPackageStep(step, checkoutOnly, depth)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 725, in _cookPackageStep
checkoutOnly, depth+1)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 501, in cook
self._cookBuildStep(step, checkoutOnly, depth)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 614, in _cookBuildStep
checkoutOnly, depth+1)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 504, in cook
self._cookPackageStep(step, checkoutOnly, depth)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 725, in _cookPackageStep
checkoutOnly, depth+1)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 501, in cook
self._cookBuildStep(step, checkoutOnly, depth)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 614, in _cookBuildStep
checkoutOnly, depth+1)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 498, in cook
self._cookCheckoutStep(step, depth)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 588, in _cookCheckoutStep
self._runShell(checkoutStep, "checkout")
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 463, in _runShell
proc = subprocess.Popen(cmdLine, cwd=step.getWorkspacePath(), env=runEnv)
File "/usr/lib/python3.4/subprocess.py", line 859, in init
restore_signals, start_new_session)
File "/usr/lib/python3.4/subprocess.py", line 1395, in _execute_child
restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory
UPLOAD work/sandbox/debian-jessie-x86_64/dist/1/workspace to d1/69/3b781f1fb99d2e1bda1c04a7a511e4f1ae5d-1.tgz
An internal Exception has occured. This should not have happenend.
Please open an issue at https://github.com/BobBuildTool/bob with the following backtrace:
Bob version 0.12.1
Traceback (most recent call last):
File "/opt/lvmvol/quellen/BobBuildTool/pym/bob/scripts.py", line 109, in catchErrors
ret = fun(*args, **kwargs)
File "/opt/lvmvol/quellen/BobBuildTool/pym/bob/scripts.py", line 180, in cmd
availableCommands[args.command][1](args.args, bobRoot)
File "/opt/lvmvol/quellen/BobBuildTool/pym/bob/scripts.py", line 29, in __build
doBuild(*args, **kwargs)
File "/opt/lvmvol/quellen/BobBuildTool/pym/bob/cmds/build.py", line 1032, in doBuild
commonBuildDevelop(parser, argv, bobRoot, False)
File "/opt/lvmvol/quellen/BobBuildTool/pym/bob/cmds/build.py", line 998, in commonBuildDevelop
builder.cook([p], p.getPackage(), args.checkout_only)
File "/opt/lvmvol/quellen/BobBuildTool/pym/bob/cmds/build.py", line 558, in cook
self._cookPackageStep(step, checkoutOnly, depth)
File "/opt/lvmvol/quellen/BobBuildTool/pym/bob/cmds/build.py", line 767, in _cookPackageStep
packageBuildId = self._getBuildId(packageStep, depth)
File "/opt/lvmvol/quellen/BobBuildTool/pym/bob/cmds/build.py", line 879, in _getBuildId
ret = step.getDigest(lambda s: self._getBuildId(s, depth+1), True)
File "/opt/lvmvol/quellen/BobBuildTool/pym/bob/input.py", line 908, in getDigest
d = calculate(arg)
File "/opt/lvmvol/quellen/BobBuildTool/pym/bob/cmds/build.py", line 879, in <lambda>
ret = step.getDigest(lambda s: self._getBuildId(s, depth+1), True)
File "/opt/lvmvol/quellen/BobBuildTool/pym/bob/cmds/build.py", line 879, in _getBuildId
ret = step.getDigest(lambda s: self._getBuildId(s, depth+1), True)
File "/opt/lvmvol/quellen/BobBuildTool/pym/bob/input.py", line 908, in getDigest
d = calculate(arg)
File "/opt/lvmvol/quellen/BobBuildTool/pym/bob/cmds/build.py", line 879, in <lambda>
ret = step.getDigest(lambda s: self._getBuildId(s, depth+1), True)
File "/opt/lvmvol/quellen/BobBuildTool/pym/bob/cmds/build.py", line 875, in _getBuildId
self.cook([step], step.getPackage(), depth)
File "/opt/lvmvol/quellen/BobBuildTool/pym/bob/cmds/build.py", line 552, in cook
self._cookCheckoutStep(step, depth)
File "/opt/lvmvol/quellen/BobBuildTool/pym/bob/cmds/build.py", line 585, in _cookCheckoutStep
False, depth+1)
File "/opt/lvmvol/quellen/BobBuildTool/pym/bob/cmds/build.py", line 558, in cook
self._cookPackageStep(step, checkoutOnly, depth)
File "/opt/lvmvol/quellen/BobBuildTool/pym/bob/cmds/build.py", line 853, in _cookPackageStep
self.__archive.uploadPackage(packageBuildId, audit, prettyPackagePath, self.__verbose)
File "/opt/lvmvol/quellen/BobBuildTool/pym/bob/archive.py", line 595, in uploadPackage
i.uploadPackage(buildId, audit, content, verbose)
File "/opt/lvmvol/quellen/BobBuildTool/pym/bob/archive.py", line 192, in uploadPackage
tar.add(content, arcname="content")
File "/opt/lvmvol/quellen/BobBuildTool/pym/bob/archive.py", line 556, in __exit__
env = { k:v for (k,v) in os.environ.items() if k in self.__whiteList }
File "/opt/lvmvol/quellen/BobBuildTool/pym/bob/archive.py", line 556, in <dictcomp>
env = { k:v for (k,v) in os.environ.items() if k in self.__whiteList }
AttributeError: 'CustomUploader' object has no attribute '_CustomUploader__whiteList'
Idea behind this is to name another yaml file on the commandline interface to setup some (e.g. platform specific) Environment variables. This can dramatically reduce the number of multiPackages or recipies. They can be more generic now.
The test/blackbox/generator
is broken:
Run blackbox test test/blackbox/generator
--- log.txt 2016-10-05 17:46:42.875836838 +0200
+++ plugins/log.txt 2016-10-05 15:09:53.920476998 +0200
@@ -1,3 +1,2 @@
->> root
- PROJECT root (g1)
+Generating project: root
root
test/blackbox/generator failed
should be possible to add a own implementation there. e.g. to get rid of the number if only one directory exists...
ok, playing around with Jenkins (Weekly release "2.19 war" from jenkins.io):
Two things I found by experiment, which should be in the docs:
bob-hash-engine
), not just the "expected" build tools; was not on my default $PATHTraceback (most recent call last):
File "<string>", line 2, in <module>
File "/home/hubert/projects/bob/pym/bob/scripts.py", line 19, in <module>
from .input import RecipeSet
File "/home/hubert/projects/bob/pym/bob/input.py", line 19, in <module>
from .pathspec import PackageSet
File "/home/hubert/projects/bob/pym/bob/pathspec.py", line 25, in <module>
import pyparsing
ImportError: No module named 'pyparsing'
I'm getting this error with the latest release of Bob when using scmOverrides on a user.yaml file:
An internal Exception has occured. This should not have happenend.
Please open an issue at https://github.com/BobBuildTool/bob with the following backtrace:
Bob version 0.11.0-60-gd33758a
Traceback (most recent call last):
File "/home/user/BobBuildTool/pym/bob/scripts.py", line 100, in catchErrors
ret = fun(*args, **kwargs)
File "/home/user/BobBuildTool/pym/bob/scripts.py", line 140, in cmd
availableCommands[verb][1](argv, bobRoot)
File "/home/user/BobBuildTool/pym/bob/scripts.py", line 32, in __develop
doDevelop(*args, **kwargs)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 1012, in doDevelop
commonBuildDevelop(parser, argv, bobRoot, True)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 974, in commonBuildDevelop
builder.cook([p], p.getPackage(), args.checkout_only)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 538, in cook
self._cookPackageStep(step, checkoutOnly, depth)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 745, in _cookPackageStep
packageBuildId = self._getBuildId(packageStep, depth)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 856, in _getBuildId
ret = step.getDigest(lambda s: self._getBuildId(s, depth+1), True)
File "/home/user/BobBuildTool/pym/bob/input.py", line 908, in getDigest
d = calculate(arg)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 856, in
ret = step.getDigest(lambda s: self._getBuildId(s, depth+1), True)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 856, in _getBuildId
ret = step.getDigest(lambda s: self._getBuildId(s, depth+1), True)
File "/home/user/BobBuildTool/pym/bob/input.py", line 908, in getDigest
d = calculate(arg)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 856, in
ret = step.getDigest(lambda s: self._getBuildId(s, depth+1), True)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 852, in _getBuildId
self.cook([step], step.getPackage(), depth)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 532, in cook
self._cookCheckoutStep(step, depth)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 553, in _cookCheckoutStep
overrides.update(scm.getActiveOverrides())
File "/home/user/BobBuildTool/pym/bob/scm/scm.py", line 48, in hash
frozenset(self.__set.items()), frozenset(self.__replaceRaw.items())))
TypeError: unhashable type: 'dict'
$ bob ls
An internal Exception has occured. This should not have happenend.
Please open an issue at https://github.com/BobBuildTool/bob with the following backtrace:
Traceback (most recent call last):
File "/home/user/BobBuildTool/pym/bob/scripts.py", line 114, in bob
availableCommands[verb][1](argv, bobRoot)
File "/home/user/BobBuildTool/pym/bob/scripts.py", line 48, in __ls
doLS(*args, **kwargs)
File "/home/user/BobBuildTool/pym/bob/cmds/misc.py", line 97, in doLS
recipes.parse()
File "/home/user/BobBuildTool/pym/bob/input.py", line 2479, in parse
self.__parse()
File "/home/user/BobBuildTool/pym/bob/input.py", line 2516, in __parse
self.__properties, self._recipeSchema):
File "/home/user/BobBuildTool/pym/bob/input.py", line 1681, in loadFromFile
for n in baseName: RECIPE_NAME_SCHEMA.validate(n)
File "/usr/local/lib/python3.4/dist-packages/schema.py", line 147, in validate
raise SchemaError('%r does not match %r' % (self, data), e)
schema.SchemaError: Regex('^[0-9A-Za-z.+-]+$') does not match 'unittests (copy)'
I accidentally executed bob in the recipes directory and got a cache-db afterwards (which shouldn't be created in that error case):
martin@kriegel ~/mib3/tsd-recipes-mib3/recipes (git)-[master] % bob dev zr3
Parse error: No recipes directory found.
martin@kriegel ~/mib3/tsd-recipes-mib3/recipes (git)-[master] % git status
Auf Branch master
Ihr Branch ist auf dem selben Stand wie 'origin/master'.
Unbeobachtete Dateien:
(benutzen Sie "git add ...", um die Änderungen zum Commit vorzumerken)
.bob-cache.shelve.db
nichts zum Commit vorgemerkt, aber es gibt unbeobachtete Dateien (benutzen Sie "git add" zum Beobachten)
martin@kriegel ~/mib3/tsd-recipes-mib3/recipes (git)-[master] %
An internal Exception has occured. This should not have happenend.
Please open an issue at https://github.com/BobBuildTool/bob with the following backtrace:
Traceback (most recent call last):
File "/home/user/BobBuildTool/pym/bob/scripts.py", line 114, in bob
availableCommands[verb][1](argv, bobRoot)
File "/home/user/BobBuildTool/pym/bob/scripts.py", line 32, in __develop
doDevelop(*args, **kwargs)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 896, in doDevelop
commonBuildDevelop(parser, argv, bobRoot, True)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 869, in commonBuildDevelop
builder.cook([p], p.getPackage(), args.checkout_only)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 487, in cook
self._cookPackageStep(step, checkoutOnly, depth)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 708, in _cookPackageStep
checkoutOnly, depth+1)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 484, in cook
self._cookBuildStep(step, checkoutOnly, depth)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 597, in _cookBuildStep
checkoutOnly, depth+1)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 487, in cook
self._cookPackageStep(step, checkoutOnly, depth)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 697, in _cookPackageStep
BobState().setInputHashes(prettyPackagePath, packageBuildId)
File "/home/user/BobBuildTool/pym/bob/state.py", line 169, in setInputHashes
self.__save()
File "/home/user/BobBuildTool/pym/bob/state.py", line 99, in __save
os.fsync(f.fileno())
OSError: [Errno 5] Input/output error
/bob dev main
WARNING: developer mode might exhibit problems and is subject to change! Use with care.
An internal Exception has occured. This should not have happenend.
Please open an issue at https://github.com/BobBuildTool/bob with the following backtrace:
Traceback (most recent call last):
File "/cygdrive/d/projects/bob/pym/bob/scrips.py", line 79, in bob
availableCommands[verb][0](recipes, argv, bobRoot)
File "/cygdrive/d/projects/bob/pym/bob/scrips.py", line 30, in __develop
doDevelop(*args, **kwargs)
File "/cygdrive/d/projects/bob/pym/bob/cmds/build.py", line 766, in doDevelop
commonBuildDevelop(recipes, parser, argv, bobRoot, True)
File "/cygdrive/d/projects/bob/pym/bob/cmds/build.py", line 728, in commonBuildDevelop
touch(sorted(rootPackages.values(), key=lambda p: p.getName()))
File "/cygdrive/d/projects/bob/pym/bob/cmds/build.py", line 668, in touch
touch([s.getPackage() for s in p.getAllDepSteps()])
File "/cygdrive/d/projects/bob/pym/bob/input.py", line 826, in getAllDepSteps
return sorted(set(self.__directDepSteps) + set(self.__indirectDepSteps))
TypeError: unsupported operand type(s) for +: 'set' and 'set'
os.glob is only working recursive in python 3.5 or higher. use os.walk instead
Changed in version 3.5: Support for recursive globs using “**”.
The hook developNamePersister
that was introduced in 6a6baa9 is not documented.
After i typed this command: 'bob ls', an exception occurred. So i open an issue at https://github.com/BobBuildTool/bob as required with the following backtrace:
Traceback (most recent call last):
File "/home/user/BobBuildTool/pym/bob/scripts.py", line 98, in bob
availableCommands[verb][1](argv, bobRoot)
File "/home/user/BobBuildTool/pym/bob/scripts.py", line 42, in __ls
from .cmds.misc import doLS
File "/home/user/BobBuildTool/pym/bob/cmds/misc.py", line 19, in
from ..input import RecipeSet, walkPackagePath
File "/home/user/BobBuildTool/pym/bob/input.py", line 412, in
class GitScm:
File "/home/user/BobBuildTool/pym/bob/input.py", line 414, in GitScm
SCHEMA = schema.Schema({
AttributeError: 'module' object has no attribute 'Schema'
When working with "bob dev" the developer expects to be able to change any of the involved sources, call bob again and it will rebuild with the changes. If binary artifacts are used and the sources are deterministic this is not true anymore. The problem is that Bob treats deterministic sources different. For these the actual sources are irrelevant and the binary artifact is downloaded just by looking at the configured tag/commit.
Steps to reproduce with bob-tutorials/binary-repository:
Instead of tripping over the #error Bob will happily download the artifact for "root":
$ bob dev --download=yes root/
>> root
DOWNLOAD dev/dist/root/1/workspace...ok
Build result is in dev/dist/root/1/workspace
Possible fixes:
You may resume at this point with '--resume' after fixing the error.
But there is no --resume arg available for bob project.
An internal Exception has occured. This should not have happenend.
Please open an issue at https://github.com/BobBuildTool/bob with the following backtrace:
Traceback (most recent call last):
File "/home/user/BobBuildTool/pym/bob/scripts.py", line 114, in bob
availableCommands[verb][1](argv, bobRoot)
File "/home/user/BobBuildTool/pym/bob/scripts.py", line 60, in __queryscm
doQuerySCM(*args, **kwargs)
File "/home/user/BobBuildTool/pym/bob/cmds/misc.py", line 176, in doQuerySCM
recipes.parse()
File "/home/user/BobBuildTool/pym/bob/input.py", line 2477, in parse
self.__cache.open()
File "/home/user/BobBuildTool/pym/bob/input.py", line 2831, in open
self.__shelve = shelve.open(".bob-cache.shelve")
File "/usr/lib/python3.4/shelve.py", line 239, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File "/usr/lib/python3.4/shelve.py", line 223, in __init__
Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
File "/usr/lib/python3.4/dbm/__init__.py", line 94, in open
return mod.open(file, flag, mode)
_gdbm.error: [Errno 11] Resource temporarily unavailable
When the default.yaml
uses include
include:
- default-local
an error is raised.
File "/data/koloska/projekte/Bob/bob/pym/bob/input.py", line 2260, in __parseUserConfig
self.__parseUserConfig(self, str(p) + ".yaml")
TypeError: __parseUserConfig() takes 2 positional arguments but 3 were given
There is a misplaced self
:
modified pym/bob/input.py
@@ -2257,7 +2257,7 @@ class RecipeSet:
self.__archive = cfg.get("archive", { "backend" : "none" })
for p in cfg.get("include", []):
- self.__parseUserConfig(self, str(p) + ".yaml")
+ self.__parseUserConfig(str(p) + ".yaml")
def __createSchemas(self):
varNameSchema = schema.Regex(r'^[A-Za-z_][A-Za-z0-9_]*$')
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "/home/hubert/projects/bob/pym/bob/__init__.py", line 55, in <module>
BOB_VERSION = getVersion()
File "/home/hubert/projects/bob/pym/bob/__init__.py", line 35, in getVersion
cwd=root, universal_newlines=True, stderr=subprocess.STDOUT)
File "/usr/lib/python3.4/subprocess.py", line 609, in check_output
output, unused_err = process.communicate(inputdata, timeout=timeout)
File "/usr/lib/python3.4/subprocess.py", line 947, in communicate
stdout = _eintr_retry_call(self.stdout.read)
File "/usr/lib/python3.4/subprocess.py", line 491, in _eintr_retry_call
return func(*args)
KeyboardInterrupt
At the moment there is no possibility to get the complete step name (e.g app/module1/module2) in the build or package step.
This can be useful, if you create module-wise-project files from bob...
On Jenkins shared packages (shared: True
) are installed into a common location. It would be nice to have that too for local builds. This feature would require a cleanup method too, though. Otherwise a user could not tell which packages in the common location are still referenced.
Hi Jan,
I merged your commits this morning. I think db49f07 brought in a regression when using multiPackage recipes. Without this commit there is only one source directory and different build / dist directories.
Now there is a separate source directory for each multiPackage package. After reverting db49f07 it's building from the same src again.
BR
Ralf
Currently there is no way to tell if an 'scmOverride' matched and what has been changed. Maybe extend "bob query-scm"?
When you add a non-root-recipe-package to jenkins with --add-root, the generated job discards old builds. I think they should persist - is it intentional or not? When the recipe has "root: True" it won't be generated with this feature.
Code line: https://github.com/BobBuildTool/bob/blob/master/pym/bob/cmds/jenkins.py#L473
It would be nice to have some words about how a sandbox should look like in the docs.
Things I found by experiment (and by peeping into basement):
etc/passwd
file containing the "nobody" user ("src/namespace-sandbox/namespace-sandbox.c:599: User not found in /etc/passwd of sandbox")home
directory ("src/namespace-sandbox/namespace-sandbox.c:480: cannot create /tmp/tmp.hSetADl61X/home/stefan: Read-only file system")tmp
directory (same)It might have helped in debugging if bob would at some level of -vvv pass -D to namespace-sandbox.
Steps to reproduce:
The problem is that Bob does not invalidate the result and input of the build step prior to building it.
When having a commit hash in a recipe git fails to checkout because of "fatal: reference is not a tree:"
The problem is the referenced line in git.py:
https://github.com/BobBuildTool/bob/blob/master/pym/bob/scm/git.py#L96
When doing "git fetch -t origin" you just fetch the tags. Instead of this it should do "git fetch origin" to get the commit references as well as the tags.
An internal Exception has occured. This should not have happenend.
Please open an issue at https://github.com/BobBuildTool/bob with the following backtrace:
Bob version 0.12.1
Traceback (most recent call last):
File "/usr/lib/python3.4/tarfile.py", line 2249, in next
tarinfo = self.tarinfo.fromtarfile(self)
File "/usr/lib/python3.4/tarfile.py", line 1083, in fromtarfile
obj = cls.frombuf(buf, tarfile.encoding, tarfile.errors)
File "/usr/lib/python3.4/tarfile.py", line 1019, in frombuf
raise EmptyHeaderError("empty header")
tarfile.EmptyHeaderError: empty header
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/user/BobBuildTool/pym/bob/scripts.py", line 109, in catchErrors
ret = fun(*args, **kwargs)
File "/home/user/BobBuildTool/pym/bob/scripts.py", line 180, in cmd
availableCommands[args.command][1](args.args, bobRoot)
File "/home/user/BobBuildTool/pym/bob/scripts.py", line 33, in __develop
doDevelop(*args, **kwargs)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 1036, in doDevelop
commonBuildDevelop(parser, argv, bobRoot, True)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 998, in commonBuildDevelop
builder.cook([p], p.getPackage(), args.checkout_only)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 558, in cook
self._cookPackageStep(step, checkoutOnly, depth)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 833, in _cookPackageStep
checkoutOnly, depth+1)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 555, in cook
self._cookBuildStep(step, checkoutOnly, depth)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 696, in _cookBuildStep
checkoutOnly, depth+1)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 558, in cook
self._cookPackageStep(step, checkoutOnly, depth)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 814, in _cookPackageStep
if self.__archive.downloadPackage(packageBuildId, audit, prettyPackagePath, self.__verbose):
File "/home/user/BobBuildTool/pym/bob/archive.py", line 600, in downloadPackage
if i.downloadPackage(buildId, audit, content, verbose): return True
File "/home/user/BobBuildTool/pym/bob/archive.py", line 154, in downloadPackage
with tarfile.open(name, "r|*", fileobj=fileobj, errorlevel=1) as tar:
File "/usr/lib/python3.4/tarfile.py", line 1578, in open
t = cls(name, filemode, stream, **kwargs)
File "/usr/lib/python3.4/tarfile.py", line 1470, in __init__
self.firstmember = self.next()
File "/usr/lib/python3.4/tarfile.py", line 2264, in next
raise ReadError("empty file")
tarfile.ReadError: empty file
some of the environment variables from defaul.yaml may not fit to the local system.
Therefore I added a include - option in the default.yaml to update the defaultEnv with the content of the given additional file.
No upload is generated for the "file" backend on Jenkins.
Executed build.sh manually and then called Bob again for the same package:
An internal Exception has occured. This should not have happenend.
Please open an issue at https://github.com/BobBuildTool/bob with the following backtrace:
Bob version 0.12.0-56-g2a4b0a1
Traceback (most recent call last):
File "/home/kloetzke/src/bob/pym/bob/scripts.py", line 115, in catchErrors
ret = fun(*args, **kwargs)
File "/home/kloetzke/src/bob/pym/bob/scripts.py", line 195, in cmd
availableCommands[args.command][1](args.args, bobRoot)
File "/home/kloetzke/src/bob/pym/bob/scripts.py", line 38, in __develop
doDevelop(*args, **kwargs)
File "/home/kloetzke/src/bob/pym/bob/cmds/build.py", line 1089, in doDevelop
commonBuildDevelop(parser, argv, bobRoot, True)
File "/home/kloetzke/src/bob/pym/bob/cmds/build.py", line 1048, in commonBuildDevelop
builder.cook([p], p.getPackage(), True if args.build_mode == 'checkout-only' else False)
File "/home/kloetzke/src/bob/pym/bob/cmds/build.py", line 577, in cook
self._cookPackageStep(step, checkoutOnly, depth)
File "/home/kloetzke/src/bob/pym/bob/cmds/build.py", line 868, in _cookPackageStep
audit = self._generateAudit(packageStep, depth, packageHash)
File "/home/kloetzke/src/bob/pym/bob/cmds/build.py", line 377, in _generateAudit
audit.addArg(os.path.join(dep.getWorkspacePath(), "..", "audit.json.gz"))
File "/home/kloetzke/src/bob/pym/bob/audit.py", line 387, in addArg
audit = Audit.fromFile(arg)
File "/home/kloetzke/src/bob/pym/bob/audit.py", line 275, in fromFile
with gzip.open(file, 'rb') as gzf:
File "/usr/lib/python3.4/gzip.py", line 52, in open
binary_file = GzipFile(filename, gz_mode, compresslevel)
File "/usr/lib/python3.4/gzip.py", line 181, in __init__
fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'dev/build/[snip]/1/workspace/../audit.json.gz'
Since the "bash-completion: handle colons in COMP_WORDBREAKS"-commit in contrib/bash-completion its broken for zsh:
martin@kriegel ~/mib3/tsd-recipes-mib3 (git)-[master] % bob dev zr3/__bob:5: command not found: _get_comp_words_by_ref
__bob:9: command not found: __ltrim_colon_completions
When trying to force build a single package with
bob build path/to/package/ -f -b -n --clean
the building of the dependencies is correctly skipped (because of -n
, though the messages state "due to --build-only" which might be OK here, too) but building of the target package is skipped, as well:
PACKAGE skipped (deterministic output in work/…/dist/1/workspace)
I guess this is not the intended behaviour of the --force
switch.
My bob version is
Bob version 0.12.0-71-gb74766d
Hi,
recently I accidentally inherited in a class the class itself and bob was running forever with no output. Maybe it would be nice if bob can detect this kind of error and exit with a proper error message. I had no clue what was happening and only discovered the error by doing a diff over my changes.
inherit: [classname]
bob ls -r
Ctrl-C
)Currently "bob clean" is only doing something in the "work" directory of release mode builds. For development mode at least two things need to be considered:
Source and/or attic directories must only be deleted if the user explicitly requested to do so.
$ bob dev --sandbox ...
...
An internal Exception has occured. This should not have happenend.
Please open an issue at https://github.com/BobBuildTool/bob with the following backtrace:
Bob version 0.11.0-60-gd33758a
Traceback (most recent call last):
File "/home/user/BobBuildTool/pym/bob/scripts.py", line 100, in catchErrors
ret = fun(*args, **kwargs)
File "/home/user/BobBuildTool/pym/bob/scripts.py", line 140, in cmd
availableCommands[verb][1](argv, bobRoot)
File "/home/user/BobBuildTool/pym/bob/scripts.py", line 32, in __develop
doDevelop(*args, **kwargs)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 1012, in doDevelop
commonBuildDevelop(parser, argv, bobRoot, True)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 974, in commonBuildDevelop
builder.cook([p], p.getPackage(), args.checkout_only)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 538, in cook
self._cookPackageStep(step, checkoutOnly, depth)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 745, in _cookPackageStep
packageBuildId = self._getBuildId(packageStep, depth)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 856, in _getBuildId
ret = step.getDigest(lambda s: self._getBuildId(s, depth+1), True)
File "/home/user/BobBuildTool/pym/bob/input.py", line 908, in getDigest
d = calculate(arg)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 856, in <lambda>
ret = step.getDigest(lambda s: self._getBuildId(s, depth+1), True)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 856, in _getBuildId
ret = step.getDigest(lambda s: self._getBuildId(s, depth+1), True)
File "/home/user/BobBuildTool/pym/bob/input.py", line 908, in getDigest
d = calculate(arg)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 856, in <lambda>
ret = step.getDigest(lambda s: self._getBuildId(s, depth+1), True)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 852, in _getBuildId
self.cook([step], step.getPackage(), depth)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 532, in cook
self._cookCheckoutStep(step, depth)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 565, in _cookCheckoutStep
False, depth+1)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 538, in cook
self._cookPackageStep(step, checkoutOnly, depth)
File "/home/user/BobBuildTool/pym/bob/cmds/build.py", line 793, in _cookPackageStep
if self.__archive.downloadPackage(packageBuildId, audit, prettyPackagePath, self.__verbose):
File "/home/user/BobBuildTool/pym/bob/archive.py", line 594, in downloadPackage
if i.downloadPackage(buildId, audit, content, verbose): return True
File "/home/user/BobBuildTool/pym/bob/archive.py", line 154, in downloadPackage
self.__extractPackage(tar, audit, content)
File "/home/user/BobBuildTool/pym/bob/archive.py", line 123, in __extractPackage
tar.extract(f, content)
File "/usr/lib/python3.4/tarfile.py", line 2019, in extract
set_attrs=set_attrs)
File "/usr/lib/python3.4/tarfile.py", line 2096, in _extract_member
self.makelink(tarinfo, targetpath)
File "/usr/lib/python3.4/tarfile.py", line 2177, in makelink
os.symlink(tarinfo.linkname, targetpath)
UnicodeEncodeError: 'ascii' codec can't encode character '\xdc' in position 65: ordinal not in range(128)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.