geni-nsf / geni-tools Goto Github PK

If a user does not have an SFA cert and omni attempts to download it, a temporary self-signed cert is created in the specified file. If the SFA cert download fails the temporary self-signed cert is left in place, but it cannot be used for any omni commands. The user is never re-prompted to download their SFA cert.

Imported from trac ticket #11, created by tmitchel on 09-27-2010 at 21:37, last modified: 09-30-2010 at 15:14

When trying to create a sliver I got:

$ src/omni.py -c ~/.gcf/omni-gec9 -a http://myplc3.gpolab.bbn.com:12346/ createsliver myplc3 myplc3-advert.ospec
INFO:omni:Loading config file /home/tmitchel/.gcf/omni-gec9
INFO:omni:Using control framework pgeni
For http://myplc3.gpolab.bbn.com:12346/ allocate = True
ERROR:omni:Error occurred. Unable to allocate from http://myplc3.gpolab.bbn.com:12346/: global name 'instanceof' is not defined.  Please run --debug to see stack trace.

Running with --debug the stacktrace is:

DEBUG:omni:Traceback (most recent call last):
  File "src/omni.py", line 391, in createsliver
    if result != None and instanceof(result, string) and result.startswith('<rspec'):
NameError: global name 'instanceof' is not defined

Imported from trac ticket #23, created by tmitchel on 10-08-2010 at 08:25, last modified: 10-08-2010 at 10:08

Omni could warn users if their sliver or slice is about to expire, or if they request a sliver renewal time past the slice expiration time. --- particularly since some frameworks will tell the user that their sliver can't expire after the slice, but not tell them when their slice expires

Imported from trac ticket #35, created by ahelsing on 05-17-2011 at 16:18, last modified: 05-17-2011 at 16:18

The certificate validity checks need to include checks for expiration. There are notes in the code that it needs to be done. PlanetLab has recently patched this and the patch needs to be integrated into gcf.

Imported from trac ticket #4, created by tmitchel on 09-20-2010 at 16:44, last modified: 09-22-2010 at 14:43

PG certs start with a text representation of the cert before the PEM representation.
openssl reads that OK.
The SFA certificate.py load_from_string should too.

1. only wrap the text in BEGIN .... END if there is NO section like that. Not if it just doesnt start with that
2. When pulling out certs, skip stuff at the beginning before the first BEGIN....

Imported from trac ticket #20, created by ahelsing on 10-04-2010 at 11:11, last modified: 10-20-2010 at 17:18

Add to omni the ability to download the SFA cert file listed in the configuration if it does not exist on disk.

Imported from trac ticket #3, created by tmitchel on 09-20-2010 at 16:00, last modified: 09-22-2010 at 14:07

Implement the renew slice capability in framework_sfa using the appropriate SFA commands.

Imported from trac ticket #15, created by tmitchel on 09-29-2010 at 09:31, last modified: 09-30-2010 at 15:15

The PG omnispec should allow the user to specify OS images to customize the nodes they allocate.

Imported from trac ticket #12, created by tmitchel on 09-27-2010 at 21:38, last modified: 09-28-2010 at 13:32

Fail early if the slice is expired, to avoid hiding the error

Imported from trac ticket #51, created by ahelsing on 08-09-2011 at 10:43, last modified: 08-09-2011 at 10:43

omni should detect that debug is enabled and not say "Please run --debug to see stack trace" if it is enabled.

Here is an example from an experimenter:

ERROR:omni:Error occurred. Unable to allocate from
https://www.emulab.net/protogeni/xmlrpc/am: <Fault 2: 'Could not map to
resources'>.  Please run --debug to see stack trace.
DEBUG:omni:Traceback (most recent call last):
 File "src/omni.py", line 322, in createsliver
   result = client.CreateSliver(urn, [slice_cred], rspec, slice_users)
 File "/usr/lib/python2.6/xmlrpclib.py", line 1199, in __call__
   return self.__send(self.__name, args)
 File "/usr/lib/python2.6/xmlrpclib.py", line 1489, in __request
   verbose=self.__verbose
 File "/usr/lib/python2.6/xmlrpclib.py", line 1253, in request
   return self._parse_response(h.getfile(), sock)
 File "/usr/lib/python2.6/xmlrpclib.py", line 1392, in _parse_response
   return u.close()
 File "/usr/lib/python2.6/xmlrpclib.py", line 838, in close
   raise Fault(**self._stack[0])
Fault: <Fault 2: 'Could not map to resources'>

Imported from trac ticket #30, created by tmitchel on 10-14-2010 at 10:17, last modified: 10-20-2010 at 15:05

If a user mistypes their ssl key password they get a lengthy stack trace that does not explain what went wrong.

Imported from trac ticket #25, created by tmitchel on 10-09-2010 at 08:38, last modified: 10-13-2010 at 15:50
CCing: ahelsing@...

For monitoring, unit testing, or richer experimenter tools, it would be nice to be able to call Omni functions as library calls

Imported from trac ticket #37, created by ahelsing on 05-17-2011 at 16:24, last modified: 05-17-2011 at 16:25

We never built a real ProtoGENI Omnispec. The first thing we can do is at least make it possible for users to allocate machines that are selected by protogeni. For instance, say "I want 3 machines running this OS image please".

Perhaps we should also add some simple links between the nodes.

Then down the road we can add more options for fancy features.

Imported from trac ticket #6, created by jkarlin on 09-23-2010 at 16:49, last modified: 09-27-2010 at 21:42
CCing: tmitchel@..., ahelsing@...

Rather than relying on redirecting STDOUT, use a -o option to redirect the listresources output to a file

Imported from trac ticket #39, created by ahelsing on 05-17-2011 at 16:42, last modified: 05-17-2011 at 16:43

If there is whitespace in the aggregates parameter in omni_config the value is not interpreted properly. This may simply be a matter of stripping whitespace after the value is split.

Imported from trac ticket #31, created by tmitchel on 10-14-2010 at 10:19, last modified: 10-20-2010 at 15:05

Coming changes to OF RSpecs (expected PM of 10/6)

• Ad Rspec specifies type="openflow"

• drop switches tag

• port_num changed to port with URN

• listing a switch means all ports on the switch. Listing just a port means just that

• don't list options with value *

  Imported from trac ticket #21, created by ahelsing on 10-04-2010 at 14:18, last modified: 10-13-2010 at 15:51

The URN of the user for installing keys has to match exactly the URN in the cert when PG is used as an AM. If the two do not match, the slice and sliver creation are successful but the user's keys are not installed at the host. If possible check user's URN against the URN at the cert and return an error to the user.

Imported from trac ticket #34, created by nriga on 04-21-2011 at 21:41, last modified: 06-17-2011 at 11:44
CCing: nriga@...

When 2 (unix) users run gcf or omni code on the same machine, you can get errors like:

  File "/home/lnevers/gcf-1.3.1-rc2/src/sfa/trust/credential.py", line 44, in <module>
     from sfa.util.sfalogging import logger
  File "/home/lnevers/gcf-1.3.1-rc2/src/sfa/util/sfalogging.py", line 120, in <module>
     info_logger = _SfaLogger(loggername='info', level=logging.INFO)
  File "/home/lnevers/gcf-1.3.1-rc2/src/sfa/util/sfalogging.py", line 54, in __init__

handler=logging.handlers.RotatingFileHandler(tmplogfile,maxBytes=1000000, backupCount=5)
  File "/usr/lib/python2.6/logging/handlers.py", line 107, in __init__
     BaseRotatingHandler.__init__(self, filename, mode, encoding, delay)
  File "/usr/lib/python2.6/logging/handlers.py", line 59, in __init__
     logging.FileHandler.__init__(self, filename, mode, encoding, delay)
  File "/usr/lib/python2.6/logging/__init__.py", line 819, in __init__
     StreamHandler.__init__(self, self._open())
  File "/usr/lib/python2.6/logging/__init__.py", line 838, in _open
     stream = open(self.baseFilename, self.mode)
 IOError: [Errno 13] Permission denied: '/tmp/sfa.log'

 lnevers@sendaria:~/gcf-1.3.1-rc2$ ls -l /tmp/sfa.log
 -rw-r--r-- 1 alice alice 0 Jul  6 14:49 /tmp/sfa.log

Imported from trac ticket #48, created by ahelsing on 07-06-2011 at 16:43, last modified: 07-08-2011 at 11:34

Add a RenewSlice capability to the gcf-ch and the framework_gcf implementation in omni now that omni has the RenewSlice command.

Imported from trac ticket #14, created by tmitchel on 09-29-2010 at 09:29, last modified: 09-29-2010 at 12:49

If a config file is specified by the -c option and it does not exist, omni does not report any warning or error. Instead it silently falls back to one of the default locations.

If the user specifies a config file that does not exist it should report the issue to the user so that it can be corrected. In this case, omni should probably exit rather than use a config file other than the one specified on the command line.

Imported from trac ticket #32, created by tmitchel on 10-28-2010 at 11:44, last modified: 11-09-2010 at 18:23

README.txt and src/geni/ch.py refer to the obsolete geni_aggregates file

Imported from trac ticket #16, created by ahelsing on 09-29-2010 at 15:00, last modified: 09-29-2010 at 15:10

createsliver, for instance, assumes that the framework returns a single credential. It should be possible for frameworks to return an array of slice credentials instead. Omni could also support loading multiple slice credential files and packing them in an array so that a user could use a set of stored credentials.

Imported from trac ticket #46, created by tmitchel on 06-23-2011 at 17:07, last modified: 08-07-2013 at 13:48

A single framework call can involve 3 XMLRPC calls - and 3 chances to mistype your PW.
Omni catches SSL PW errors at the top level.
The result is you may have to redo some XMLRPC calls, and have more chances to mistype.

Refactor _do_ssl to be a library function that the frameworks call directly

Imported from trac ticket #29, created by ahelsing on 10-13-2010 at 15:51, last modified: 05-09-2011 at 09:23

listresources takes an rspec_version option with a type and a version, allowing you to request what RSpec type and version you want. Allow the user to supply that

Imported from trac ticket #40, created by ahelsing on 05-19-2011 at 16:38, last modified: 05-19-2011 at 16:40

We should support saving and loading a slice credential from a file. This allows passing around delegated credentials, for example

Imported from trac ticket #41, created by ahelsing on 05-20-2011 at 11:15, last modified: 05-20-2011 at 11:54

Omni currently requires a name parameter on each user in the config file. This parameter is not part of the AM API, and is not used by any existing aggregate manager. It should be removed.

Imported from trac ticket #9, created by tmitchel on 09-27-2010 at 21:30, last modified: 09-30-2010 at 16:41

Currently we generate 1 flowspace per switch-port when converting omnispecs to RSpecs for OpenFlow. We should instead group these such that idential (related?) flowspaces are combined, and just list multiple switches/ports.

Imported from trac ticket #19, created by ahelsing on 10-04-2010 at 09:17, last modified: 12-13-2011 at 14:21

SFA has renamed its calls (e.g. get_user_credential -> GetUserCredential) and reordered some of the parameters.

Omni needs to reflect these changes.

On old sites (such as PLC) the old functions remain (were never directly uninstalled) and so omni still works on such sites. On fresh installs of myplc, omni will not work until we fix this.

Imported from trac ticket #22, created by jkarlin on 10-07-2010 at 16:36, last modified: 10-29-2010 at 11:10
CCing: tmitchel@..., ahelsing@...

Now that you can save slice credentials and supply those, a utility to delegate slice credentials would be nice

Imported from trac ticket #44, created by ahelsing on 06-23-2011 at 14:16, last modified: 07-06-2011 at 09:06

If the following are true, it is possible to download the wrong PlanetLab gid in Omni causing Omni to not work:

1. the user is a PI
2. the user has a slice with the same name as their username

Under these circumstances the user and the slice have the same hrn (eg plc.bbn.smith). If a PI calls Resolve with this hrn, it returns a list of length > 1. The code returns the first item in the list. The result is that if you don't have a user gid downloaded, when omni downloads it for you it won't work.

Imported from trac ticket #49, created by sedwards on 07-12-2011 at 09:04, last modified: 07-14-2011 at 14:26
CCing: ahelsing@..., tmitche@..., vthomas@...

Trying to allocate a PLC resource with a PG credential is not working, it seems like the PLC code used to create a new user on-the-fly is causing problems with the PLC database. Note, that this same problem occurs if you try to allocate from PLE using PLC so this is not an emulab specific bug, it's a problem with handling foreign users/slices.

jkarlin@ubu:/dev/gcf/src$ ./omni.py -f pg createsliver tg res
INFO:omni:Using control framework pg
For https://www.emulab.net/protogeni/xmlrpc/am allocate = True
For http://www.planet-lab.org:12346 allocate = True
{u'https://www.emulab.net/protogeni/xmlrpc/am': '', u'http://www.planet-lab.org:12346': 'Technische Universitaet Dresdenplanet1.inf.tu-dresden.de'}
ERROR:omni:Error occurred. Unable to allocate from https://www.emulab.net/protogeni/xmlrpc/am: <Fault 2: 'Could not map to resources'>. Please run --debug to see stack trace.
ERROR:omni:Error occurred. Unable to allocate from http://www.planet-lab.org:12346: <Fault 111: "Internal API error: <Fault 102: 'person_id 1: AddSite: Invalid argument: Login base must consist only of lowercase ASCII letters or numbers'>">. Please run --debug to see stack trace.
jkarlin@ubu:/dev/gcf/src$

Imported from trac ticket #7, created by jkarlin on 09-27-2010 at 09:36, last modified: 09-27-2010 at 10:31
CCing: tmitchel@..., ahelsing@...

If OF has no resources it gives an empty rspec:

<rspec><network location="gpo" name="masada"><switches /><links /></network></rspec>

But rspec_of refuses to translate that, so you get an Exception in omni.
Fix the rspec_of to handle that.

Imported from trac ticket #10, created by tmitchel on 09-27-2010 at 21:32, last modified: 10-13-2010 at 15:49

SFA's credential changed a bit. It no longer has set_lifetime and now has set_expiration. There is only one place in GCF code that needed adjustment (cred_util.py).

Imported from trac ticket #24, created by jkarlin on 10-08-2010 at 15:14, last modified: 10-08-2010 at 15:14

The PG omnispec currently only allows nodes (compute resources). Add the capability to add links between nodes (topologies).

Imported from trac ticket #13, created by tmitchel on 09-27-2010 at 21:39, last modified: 06-17-2011 at 11:33

Certificates must be explicitly set to be version 3 when they have extensions. Default is V1 but wrong

Imported from trac ticket #36, created by ahelsing on 05-17-2011 at 16:22, last modified: 05-17-2011 at 16:23

Some clearinghouses allow you to list all slices registered to a user (PG). Support that.

Imported from trac ticket #38, created by ahelsing on 05-17-2011 at 16:40, last modified: 05-17-2011 at 16:41

XML RPC Faults are ugly.
They have embedded \ns -- part of a Traceback.
Print them much more cleanly

Imported from trac ticket #28, created by ahelsing on 10-13-2010 at 15:49, last modified: 10-21-2010 at 13:30

A user reported that omni did not abort after she had failed to type in her private key password 3 times. When a user types in their password incorrectly 3 times, the entire process should probably be aborted.

The following log demonstrates the problem:

$ ./src/omni.py renewslice my_test 20111130T18:00:00 -c ~/.gcf/omni_config_plc
INFO:omni:Loading config file /home/user/.gcf/omni_config_plc
INFO:omni:Using control framework plc
INFO:omni.sfa:SFA Registry: http://www.planet-lab.org:12345
INFO:omni.sfa:SFA Slice Manager: http://www.planet-lab.org:12347
Enter PEM pass phrase:
ERROR:omni.sfa:Wrong pass phrase for private key.
INFO:omni.sfa:.... please retry.
Enter PEM pass phrase:
ERROR:omni.sfa:Wrong pass phrase for private key.
ERROR:omni.sfa:Wrong pass phrase after 2 tries. Cannot do Get SFA user credential from registry http://www.planet-lab.org:12345 for user plc.bbn.user using cert file /home/user/.gcf/plc.bbn.user.gid.
ERROR:omni.sfa:Cannot get a slice credential without a user credential
ERROR:omni.sfa:Failed to renew slice urn:publicid:IDN+plc:bbn+slice+my_test: could not get a slice credential
WARNING:omni:Failed to renew slice my_test
Enter PEM pass phrase:
Enter PEM pass phrase:
INFO:omni:Slice urn:publicid:IDN+plc:bbn+slice+my_test expires on 2011-11-30 17:00:00 UTC
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed renewslice:

Options as run:
      configfile: /home/user/.gcf/omni_config_plc
      framework: plc
      native: True

Args: renewslice my_test 20111130T18:00:00

Result Summary: Failed to renew slice my_test
Slice urn:publicid:IDN+plc:bbn+slice+my_test expires on 2011-11-30 17:00:00 UTC
INFO:omni: ============================================================

Imported from trac ticket #43, created by tmitchel on 06-17-2011 at 10:04, last modified: 07-06-2011 at 09:07

The src/gen-certs.py script silently ignores the argument to the -d/--directory command line option.

Imported from trac ticket #50, created by tmitchel on 07-15-2011 at 15:41, last modified: 10-17-2014 at 16:24

Omni should log config file used, and document search path in omni_config and README-omni (and the wiki)

Imported from trac ticket #17, created by ahelsing on 09-29-2010 at 15:02, last modified: 09-29-2010 at 15:10

Starting with SFA 1.0.24, the SFA / Planetlab RSpec format has changed.

Omni no longer parses those correctly into omnispecs

Imported from trac ticket #45, created by ahelsing on 06-23-2011 at 15:06, last modified: 12-13-2011 at 14:22

AM API specifies you should supply a timezone in renewsliver.

AM API credential format says you don't have to specify a timezone.

If you aren't careful, you end up comparing times with a timezone to those without - sometimes in a local time and sometimes in UTC.

Original symptom:

When trying to renewslice at ProtoGENI:

$ src/omni.py -c ~/.gcf/omni-utah -a https://boss.emulab.net/protogeni/xmlrpc/am renewslice tmgec11 20110801T17:00
INFO:omni:Loading config file /home/tmitchel/.gcf/omni-utah
INFO:omni:Using control framework pg
Enter PEM pass phrase:
INFO:omni.protogeni:Requesting new slice expiration '2011-08-01T17:00:00'
INFO:omni:Slice tmgec11 now expires at 2011-08-01 17:00:00 UTC
Traceback (most recent call last):
 File "src/omni.py", line 1966, in <module>
   sys.exit(main())
 File "src/omni.py", line 1960, in main
   API_call(framework, config, args, opts, verbose=opts.verbose)
 File "src/omni.py", line 1787, in API_call
   result = handler._handle(args)
 File "src/omni.py", line 121, in _handle
   return getattr(self,call)(args[1:])
 File "src/omni.py", line 1252, in renewslice
   retVal +=self._print_slice_expiration(urn)
 File "src/omni.py", line 1455, in _print_slice_expiration
   if sliceexp <= now:
TypeError: can't compare offset-naive and offset-aware datetimes

Imported from trac ticket #47, created by tmitchel on 06-30-2011 at 15:02, last modified: 07-06-2011 at 09:50
CCing: ahelsing@..., sedwards@...

document that Omni users probably want to do the quick start to ensure they have SW dependencies

document how Expedient AMs can trust other CHs

document sample Expedient AM URN and URL

Imported from trac ticket #18, created by ahelsing on 09-29-2010 at 15:05, last modified: 09-29-2010 at 15:09

If a user defaults their controller url to localhost, it can lead to cycles and crashing. Currently omnispec handles openflow rspecs by filling it default values for all of the possible paramters, and the default for controller url is localhost:6633. Change this to something innocuous.

Imported from trac ticket #5, created by jkarlin on 09-22-2010 at 09:38, last modified: 09-22-2010 at 09:38

listresources on an Openflow AM returns a resv_rspec whose format is different than the advertisement rspec -- so it doesn't know how to parse it.

The result is that you'll get an error - and it will print out in pretty XML the returned resv_rspec. Which isn't a bad option.

To fix this the easiest thing is to create a resv_rspec_of file that understands that format or something like that.

Imported from trac ticket #26, created by ahelsing on 10-11-2010 at 20:49, last modified: 12-13-2011 at 14:21

Expired user certificates generate ugly SSL errors. Tell user the issue and how to fix it.

Imported from trac ticket #52, created by ahelsing on 08-09-2011 at 10:44, last modified: 08-09-2011 at 10:45

Manifest RSpec returns from createsliver are worth saving

Use the -o option to allow that

Imported from trac ticket #42, created by ahelsing on 05-23-2011 at 14:50, last modified: 05-23-2011 at 16:58

SFA does some name mangling to slices to create a login name on the allocated virtual machines. OMNI guesses what the name mangling will be and informs the user.

We should remove the code that guesses from omni, and have the user call 'sliverstatus' to find out what their login name is.

Imported from trac ticket #27, created by jkarlin on 10-12-2010 at 15:17, last modified: 10-20-2010 at 13:43
CCing: tmitchel@..., ahelsing@...

Ezra Kissel submitted an omni patch for requesting specific ProtoGENI nodes. Incorporate that patch.

Imported from trac ticket #33, created by tmitchel on 01-27-2011 at 21:56, last modified: 01-28-2011 at 08:45

PG doesn't seem to like PLC's user credential in listresources (note that getversion works).

According to Tom the following error occurs when the SSL cert's URN does not match the credential's owner urn. Will dig into that.

jkarlin@ubu:~/dev/gcf/src$ ./omni.py -f plc listresources > res
INFO:omni:Using control framework plc
INFO:omni.sfa:SFA Registry: http://www.planet-lab.org:12345
INFO:omni.sfa:SFA Slice Manager: http://www.planet-lab.org:12347
ERROR:omni:Failed to List Resources from https://www.emulab.net/protogeni/xmlrpc/am (https://www.emulab.net/protogeni/xmlrpc/am): <Fault 3: 'This is not your credential'>

Imported from trac ticket #8, created by jkarlin on 09-27-2010 at 09:40, last modified: 09-27-2010 at 10:31
CCing: tmitchel@..., ahelsing@...