leerssej / freebase-python Goto Github PK

What steps will reproduce the problem?
1. do the first code example in the README
2. freebase.mqlread(query)
3. for album in result.album:
    print album
4. NameError: name 'result' is not defined

What is the expected output? What do you see instead?

I expected to see the list of albums, I got an error.

What version of the product are you using? On what operating system?

1.06

Please provide any additional information below.


freebase.mqlread(query)
should be 

result = freebase.mqlread(query)

What steps will reproduce the problem?
1. Create a write query you know will fail 
2. mqlwrite(query)
3. Freebase will return a status 400 without a Request body

What is the expected output? What do you see instead?

_raise_service_error does not fail, and mqlwrite raises a MetawebError.


What version of the product are you using? On what operating system?

1.6.0

Please provide any additional information below.

Here is how I have fixed the bug in locally.  See line 88 of httpclients.py.  
This fix will accommodate scenarios where no body is returned, but still 
properly raise the MetawebError. 


if resp.status_code != 200:
  try:
    body = resp.body
  except AttributeError:
    body = "" 
  self._raise_service_error(url, resp.status_code, 
                                      resp.headers['content-type'], body)

The parameters mql_input, mql_output, and mql_filter on the geosearch
method don't work unless they're pre-serialized.  This is different than
the way the other methods such as mqlread are specified, so I don't think
it's intentional.  It's definitely undesirable.

The attached patch will serialize these parameters correctly.

Index: session.py
===================================================================
--- session.py  (revision 217)
+++ session.py  (working copy)
@@ -814,7 +814,7 @@
         if location_type:
            form["location_type"] = location_type
         if mql_input:
-           form["mql_input"] = mql_input
+           form["mql_input"] = json.dumps(mql_input, separators=SEPARATORS)
         if limit:
            form["limit"] = limit
         if start:
@@ -826,7 +826,7 @@
         if intersect:
            form["intersect"] = intersect
         if mql_filter:
-           form["mql_filter"] = mql_filter
+           form["mql_filter"] = json.dumps(mql_filter, separators=SEPARATORS)
         if within:
            form["within"] = within
         if inside:
@@ -838,7 +838,7 @@
         if format:
            form["format"] = format
         if mql_output:
-           form["mql_output"] = mql_output
+           form["mql_output"] = json.dumps(mql_output, separators=SEPARATORS)

         if format == "json":
             r = self._httpreq_json(service, 'POST', form=form)

The freebase package is missing from the index & search on PyPI.  It can still 
be installed using 

  pip install freebase

still exists at http://pypi.python.org/pypi/freebase/1.0.8 but 
http://pypi.python.org/pypi/freebase/ (which is linked to from here) gives an 
error.

Line path 649 in cookielib.py: 
urlib.quote(path, HTTP_PATH_SAFE)

Should be: 
urllib.quote(path, HTTP_PATH_SAFE)


Surprised no one has come across this.  Nasty bug when using freebase.login().

What steps will reproduce the problem?
1. download and install trunk
2. do not install/remove python-httplib2 (ubuntu 8.04)
3. run the test suite

What is the expected output? What do you see instead?
expected: all tests passed.
got: 

======================================================================
ERROR: test_login (__main__.TestFreebase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_all.py", line 45, in test_login
    mss.login()
  File "/usr/lib/python2.5/site-packages/freebase/api/session.py", line
392, in login
    raise MetawebError("login error: %s", e)
MetawebError: ('login error: %s', AttributeError("'Urllib2Client' object
has no attribute '_raise_service_error'",))


What version of the product are you using? On what operating system?

trunk, Ubuntu 8.04

Please provide any additional information below.

as reported by Kurt Bollacker:

The publicly available freebase-api python module has a memory leak when
using mqlreaditer() for a long running query. It only takes a couple hours
to fill 4GB of RAM. For a comparison test, I added a similar generator
functionality to readall() in metaweb.py, but the problem does not exist
there. This suggests that the bug is specific to freebase-api and not just
cursoring python with generators.

The freebase.search() function is missing some parameters -- specifically 
type_exclude at http://www.freebase.com/docs/web_services/search#limiting, but 
there may be others. Since passing extra parameters to Freebase APIs doesn't 
seem to do any harm, I suggest this definition of search():

@json_params
def search(self, query, **kwargs):
    service = "/api/service/search"
    form = dict(query=query)
    form.update(kwargs)
    return self._mqlresult(self._httpreq_json(service, 'POST', form=form))

This has the advantage that further parameters added in the future will 
continue to work. 

What steps will reproduce the problem?
1. Request this query 
[{'c:key': [{'namespace': '/wikipedia/en_id', 'value': None}], 'a:key': 
[{'namespace': '/wikipedia/en_id', 'value': None}], 'b:key': [{'namespace': 
'/wikipedia/en', 'value': None}], 'name': {'lang': '/lang/en', 'value': None}, 
'type|=': ['/people/person']}]


What is the expected output? What do you see instead?
Results returned. It seems freebase require special Httplib2Client

What version of the product are you using? On what operating system?
trunk, Ubuntu 12.04

Please provide any additional information below.

[{'c:key': [{'namespace': '/wikipedia/en_id', 'value': None}], 'a:key': 
[{'namespace': '/wikipedia/en_id', 'value': None}], 'b:key': [{'namespace': 
'/wikipedia/en', 'value': None}], 'name': {'lang': '/lang/en', 'value': None}, 
'type|=': ['/people/person']}]
2013-05-14 11:38:20 INFO session.py: 397 POST 
http://api.freebase.com/api/service/mqlread
FORM:
  query={"cursor":true,"query":[{"c:key":[{"namespace":"/wikipedia/en_id","value":null}],"a:key":[{"namespace":"/wikipedia/en_id","value":null}],"b:key":[{"namespace":"/wikipedia/en","value":null}],"name":{"lang":"/lang/en","value":null},"type|=":["/people/person"]}],"escape":false}
HEADERS:
  content-type: application/x-www-form-urlencoded; charset=utf-8
  x-metaweb-request: Python
  user-agent: python freebase.api-1.0.4
Traceback (most recent call last):
  File "freebase_entities.py", line 105, in <module>
    main(args)
  File "freebase_entities.py", line 75, in main
    dump_results(q)
  File "freebase_entities.py", line 57, in dump_results
    for row in r:
  File "/usr/local/lib/python2.7/dist-packages/freebase/api/session.py", line 569, in mqlreaditer
    r = self._httpreq_json(service, 'POST', form=dict(query=qstr))
  File "/usr/local/lib/python2.7/dist-packages/freebase/api/session.py", line 420, in _httpreq_json
    resp, body = self._httpreq(*args, **kws)
  File "/usr/local/lib/python2.7/dist-packages/freebase/api/session.py", line 406, in _httpreq
    return self._http_request(url, method, body, headers)
  File "/usr/local/lib/python2.7/dist-packages/freebase/api/httpclients.py", line 65, in __call__
    self.log.error('SOCKET FAILURE: %s', e.fp.read())
AttributeError: 'Httplib2Client' object has no attribute 'log'


> 
/usr/local/lib/python2.7/dist-packages/freebase/api/httpclients.py(65)__call__()
-> self.log.error('SOCKET FAILURE: %s', e.fp.read())

You need to add 
self.log = logging.getLogger("freebase")

to the __init__ of Httplib2Client

What steps will reproduce the problem?

This is a somewhat rare issue, but I've seen it at least five times. It
seems to be a typical trying-to-urlquote-some-undecoded-unicode type problem.

Here's the traceback:

  File "/freebase/api/session.py", line 144, in urlencode_weak
    return urlquote(s, safe=',/:$')
  File "/base/python_dist/lib/python2.5/urllib.py", line 1205, in quote
    res = map(safe_map.__getitem__, s)
KeyError: u'\u2019'


What version of the product are you using? On what operating system?

1.02


I tried fixing this using a smart_unicode type method to properly encode s
before urlquoting it, but I couldn't do it in a few minutes. Might try
again later. 

What steps will reproduce the problem?
1. Download and install trunk
2. Install python-httplib2 (on ubuntu)
3. Edit session.py, change line 391 to say "except Exception, e:"
4. Run the test suite

What is the expected output? What do you see instead?

Expected "all tests passed"
Got:

ERROR: test_write (__main__.TestFreebase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_all.py", line 74, in test_write
    mss.login()
  File "/usr/lib/python2.5/site-packages/freebase/api/session.py", line
392, in login
    raise MetawebError("login error: %s", e)
MetawebError: ('login error: %s', AttributeError("'module' object has no
attribute 'HttpLib2ErrorWithResponse'",))

try again in the Python interpreter:

Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52) 
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import httplib2
>>> httplib2.HttpLib2ErrorWithResponse
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'HttpLib2ErrorWithResponse'
>>>

What version of the product are you using? On what operating system?

trunk on Ubuntu 8.04

Please provide any additional information below.

This may be a documentation issue. If you require a particular version of
httplib2 (i.e. not the one that comes with Ubuntu 8.04), you may get that
particular Exception, which I see does actually exist in httplib2's trunk.

When there is a socket error in httplib2Client, it tries to log a message using 
self.log which is not initialized. I'll attach my patch file even thought it's 
trivial.

This appears to have been fixed in r253, but no new release has been made.  I 
guess I've been using a private copy, but I just installed Python 2.7 and a 
fresh easy_installed copy of the freebase egg and now I'm getting my output 
polluted with all these prints saying "sending..." for every query.

Could someone upload a new release please?

I should be able to invoke freebase-python with some kind of descriptive
information about where my app lives, what it's name is, who to contact, etc.

This information is incredibly valuable when bad apps go awry and we're
trying to find them (the descriptive information should be included in the
user agent field during HTTP sessions).

I ran the latest freebase SVN code (r56) and receive the following errors
when I run test_all.py:

$ python test_all.py 
No handlers could be found for logger "root"
EFE
======================================================================
ERROR: test_login (__main__.TestFreebase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_all.py", line 45, in test_login
    mss.login()
  File
"/home/joseph/utils/lib/python2.6/site-packages/freebase-0.2.5dev_r56-py2.6.egg/
freebase/api/session.py",
line 390, in login
    password=self.password))
  File
"/home/joseph/utils/lib/python2.6/site-packages/freebase-0.2.5dev_r56-py2.6.egg/
freebase/api/session.py",
line 335, in _httpreq_json
    resp, body = self._httpreq(*args, **kws)
  File
"/home/joseph/utils/lib/python2.6/site-packages/freebase-0.2.5dev_r56-py2.6.egg/
freebase/api/session.py",
line 321, in _httpreq
    return self._http_request(url, method, body, headers)
  File
"/home/joseph/utils/lib/python2.6/site-packages/freebase-0.2.5dev_r56-py2.6.egg/
freebase/api/httpclients.py",
line 41, in __call__
    self._raise_service_error(url, e.code, e.info().type, e.fp.read())
  File
"/home/joseph/utils/lib/python2.6/site-packages/freebase-0.2.5dev_r56-py2.6.egg/
freebase/api/session.py",
line 332, in _raise_service_error
    raise MetawebError, 'request failed: %s: %r %r' % (url, status, body)
MetawebError: request failed:
http://sandbox.freebase.com/api/account/login: 400 '{\n  "code":
"/api/status/error/account/invalid",\n  "messages": [\n    {\n      "code":
"/api/status/error/input/invalid",\n      "info": {\n        "value":
null\n      },\n      "message": "Invalid username or password."\n    }\n 
],\n  "status": "400 Bad Request",\n  "transaction_id":
"cache;cache02.sandbox.sjc1:8101;2009-05-28T17:56:43Z;0001"\n}'

======================================================================
ERROR: test_write (__main__.TestFreebase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_all.py", line 74, in test_write
    mss.login()
  File
"/home/joseph/utils/lib/python2.6/site-packages/freebase-0.2.5dev_r56-py2.6.egg/
freebase/api/session.py",
line 390, in login
    password=self.password))
  File
"/home/joseph/utils/lib/python2.6/site-packages/freebase-0.2.5dev_r56-py2.6.egg/
freebase/api/session.py",
line 335, in _httpreq_json
    resp, body = self._httpreq(*args, **kws)
  File
"/home/joseph/utils/lib/python2.6/site-packages/freebase-0.2.5dev_r56-py2.6.egg/
freebase/api/session.py",
line 321, in _httpreq
    return self._http_request(url, method, body, headers)
  File
"/home/joseph/utils/lib/python2.6/site-packages/freebase-0.2.5dev_r56-py2.6.egg/
freebase/api/httpclients.py",
line 41, in __call__
    self._raise_service_error(url, e.code, e.info().type, e.fp.read())
  File
"/home/joseph/utils/lib/python2.6/site-packages/freebase-0.2.5dev_r56-py2.6.egg/
freebase/api/session.py",
line 332, in _raise_service_error
    raise MetawebError, 'request failed: %s: %r %r' % (url, status, body)
MetawebError: request failed:
http://sandbox.freebase.com/api/account/login: 400 '{\n  "code":
"/api/status/error/account/invalid",\n  "messages": [\n    {\n      "code":
"/api/status/error/input/invalid",\n      "info": {\n        "value":
null\n      },\n      "message": "Invalid username or password."\n    }\n 
],\n  "status": "400 Bad Request",\n  "transaction_id":
"cache;cache01.sandbox.sjc1:8101;2009-05-28T17:56:44Z;0003"\n}'

======================================================================
FAIL: test_read (__main__.TestFreebase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_all.py", line 63, in test_read
    self.assertEqual( '/guid/9202a8c04000641f8000000000092a01',
result['id'][0]['value'], str(result['id']))
AssertionError: [{'type': '/type/id', 'value': '/en/sting'}]

----------------------------------------------------------------------
Ran 3 tests in 1.382s

FAILED (failures=1, errors=2)

What steps will reproduce the problem?
1. import logging and define some handlers
2. import freebase and do some queries

What is the expected output? What do you see instead?

There's tons of debugging output from the root logger, which is used (i
think) everywhere in the libraray. Using a named logger would solve the
issue. I.e. logging.getLogger("name") instead of logging.getLogger()

Thanks,
Sergei.

I don't know if experimental APIs qualify for inclusion, but since the
geosearch API is supported, I'm guessing they do.

It'd be nice to have support for 

http://www.freebase.com/experimental/topic/{mode}?id={freebase_id}&domains={doma
in_list}

http://www.freebase.com/docs/topic_api

What steps will reproduce the problem?
1. Download most recent code
2. Run examples/get_album.py

What is the expected output? What do you see instead?

Expected output is a list of Beatles tracks, but instead is the error: "Type 
/music/album does not have property track"

What version of the product are you using? On what operating system?

Subversion r278 (current 2011/03/07) on python 2.6.5 on cygwin under Vista.

Please provide any additional information below.

The example does not use the schema with tracks separated from albums through 
releases.

A corrective patch is attached.

What steps will reproduce the problem?

Run the code found on the getting started page for mqlreadmulti, specifically:

r = freebase.mqlreadmulti([
      {"id":"/en/the_beatles", "name":None}, 
      {"id":"/en/the_police", "name":None}
    ])

What is the expected output? What do you see instead?
An expected query would be straightforward and look like this:

queries = {
'q0':
  {'query': [{"id":"/en/the_beatles", "name":None}] },
'q1':
  {'query': [{"id":"/en/the_police", "name":None}] }
}

Try the link here:
https://api.freebase.com/api/service/mqlread?queries={%22q0%22:{%22query%22:[{%2
2id%22:%22/en/the_beatles%22,%22name%22:null}]},%22q1%22:{%22query%22:[{%22id%22
:%22/en/the_police%22,%22name%22:null}]}}

Instead, the python API generates this query, note the multiple nesting and 
double entry of "the_beatles":

"q1": {
    "q0": {
      "query": {
        "id": "/en/the_beatles", 
        "name": null
      }, 
      "escape": false
    }, 
    "escape": false, 
    "query": {
      "id": "/en/the_police", 
      "name": null
    }
  }, 
  "q0": {
    "query": {
      "id": "/en/the_beatles", 
      "name": null
    }, 
    "escape": false
  }

What version of the product are you using? On what operating system?
Freebase 1.0.6 on Mac OSX

Please provide any additional information below.
I am new to Freebase and MQL, so I am fully aware that I may be missing some 
complexity, so I apologize in advance if that is the case.

Thanks,
Austin

PS - Here is the full console output:

INFO     2011-06-19 15:05:46,010 session.py:650] /api/service/mqlread: 
  "q1": {
    "q0": {
      "query": {
        "id": "/en/the_beatles", 
        "name": null
      }, 
      "escape": false
    }, 
    "escape": false, 
    "query": {
      "id": "/en/the_police", 
      "name": null
    }
  }, 
  "q0": {
    "query": {
      "id": "/en/the_beatles", 
      "name": null
    }, 
    "escape": false
  }
INFO     2011-06-19 15:05:46,011 session.py:419] POST 
http://api.freebase.com/api/service/mqlread
FORM:
  queries={"q1":{"q0":{"query":{"id":"/en/the_beatles","name":null},"escape":false},"escape":false,"query":{"id":"/en/the_police","name":null}},"q0":{"query":{"id":"/en/the_beatles","name":null},"escape":false}}
HEADERS:
  x-requested-with: Freebase-Python
  content-type: application/x-www-form-urlencoded; charset=utf-8
  user-agent: python freebase.api-1.0.4
INFO     2011-06-19 15:05:46,470 session.py:657] /api/service/mqlread result: {
  "q1": {
    "code": "/api/status/ok", 
    "result": {
      "id": "/en/the_police", 
      "name": "The Police"
    }
  }, 
  "q0": {
    "code": "/api/status/ok", 
    "result": {
      "id": "/en/the_beatles", 
      "name": "The Beatles"
    }
  }, 
  "code": "/api/status/ok", 
  "status": "200 OK", 
  "transaction_id": "cache;cache01.p01.sjc1:8101;2011-06-19T15:05:46Z;0025"
}
INFO     2011-06-19 15:05:46,471 session.py:481] result: 
  "code": "/api/status/ok", 
  "result": {
    "id": "/en/the_beatles", 
    "name": "The Beatles"
  }
INFO     2011-06-19 15:05:46,471 session.py:481] result: 
  "code": "/api/status/ok", 
  "result": {
    "id": "/en/the_police", 
    "name": "The Police"
  }

What steps will reproduce the problem?
1. Go to http://freebase-images.appspot.com/
2. Type in any word, and enter.
3. Nothing happens.

What is the expected output? What do you see instead?
I expected a search result for images.  But nothing happens.


What version of the product are you using? On what operating system?

Latest.

Please provide any additional information below.

What steps will reproduce the problem?
1. Download and install the latest freebase-api on ubuntu
2. Install python-httplib2
3. run the test suite

What is the expected output? What do you see instead?

I expected: All tests passed.
I got: 

Traceback (most recent call last):
  File "test_all.py", line 45, in test_login
    mss.login()
  File "/usr/lib/python2.5/site-packages/freebase/api/session.py", line
391, in login
    except urllib2.HTTPError, e:
NameError: global name 'urllib2' is not defined

What version of the product are you using? On what operating system?

trunk, on Ubuntu 8.04

Please provide any additional information below.

This is fixable by changing the except statement on line 391 of session.py
to look for Exception, but that's not a good fix. 

Right now, freebase.blurb(id) returns a string.  I think it'd be more helpful 
if the decoding were done on the string and the function returns unicode. 

The library needs to be updated to support the new Google APIs.  The home page 
has said this is coming for months, but we're now weeks away from the 
retirement of the old APIs, so the need is becoming much more urgent.

If you look at the attached picture of the SVN revision graph, you can see that 
the 1.0.7 and 1.0.8 releases were made off a branch of the trunk which 
*predates* the 1.0.6 release.

This has caused a bunch of functionality to be deleted from the API such as the 
escape and envelope parameters on mqlreaditer() creating an API which is 
incompatible.

Additionally, the trunk is behind both the 1.0.6 and 1.0.8 branches.  The diffs 
from both branches need to be merged back into the trunk and a new 1.0.9 
release created.