tanuki / tanuki.py Goto Github PK
View Code? Open in Web Editor NEWPrompt engineering for developers
Home Page: https://pypi.org/project/tanuki.py/
License: MIT License
Prompt engineering for developers
Home Page: https://pypi.org/project/tanuki.py/
License: MIT License
Currently only Uvicorn is supported, and therefore this project doesn't support multiple worker threads. We should add support for Gunicorn to support that popular usecase.
PDF classifier for freight brokerage company
Is your feature request related to a problem? Please describe.
We want to generalise the package by supporting local OSS models as students or teachers
Describe the solution you'd like
We need to support OSS models (Zephyr, Mistral, LLama, Qwen etc) as teacher or student. Currently we are only relying on OpenAI, whereas the library should be generalised to local OSS models
We should avoid passing loosely typed dictionaries around if we can avoid it. This should be replaced by a Pydantic class.
Note: This is in BufferedLogger
.
Input: Stream of Discord messages with the time + user ID
Output: Alert if promotional content, entity / product being promoted, user ID
Purpose: To flag to admins and moderators if content is very promotional or about selling stuff vs. genuine dialogue.
This currently errors out during parsing of aligns
class Location(BaseModel):
"""A representation of a US city and state"""
city: str = Field(description="The city's proper name")
state: str = Field(description="The state's two-letter abbreviation (e.g. NY)")
@Monkey.patch
def construct_object_1(input: str) -> Location:
"""
Using the input, construct the output model in the type hint
"""
@Monkey.align
def align():
assert construct_object_1("The Windy City") == Location(city="Chicago", state="IL")
Reason is it is saved as this into the align dataset
"{'args': ('The Windy City',), 'kwargs': {}, 'output': Location(city='Chicago', state='IL')}\r"
LLM providers, specifically OpenAI, support streamed responses. We should support this.
Requirements:
We are currently running 0.28.1 and should upgrade.
Is your feature request related to a problem? Please describe.
We want to generalise the package by supporting AWS Bedrock models as students or teachers (Claude2 for instance)
Describe the solution you'd like
We need to support AWS Bedrock models as teacher or student. Currently we are only relying on OpenAI, whereas the library should be more generalised
Currently, we don't know where our users are having issues. We should intercept exceptions that are not caused by third parties (i.e OpenAI) and submit them to the endpoint along with a truncated stack-trace (not including the names of any files on the users machine).
This will help us debug any user issues.
We should support asynchronous processing of patched function like as follows:
@monkey.patch
async def iterate_presidents() -> Iterable[str]:
""""List the presidents of the United States""""
@monkey.patch
async def tell_me_more_about(topic: str) -> str:
"""Describe this in more detail"
# For each president listed, generate a description concurrently
start_time = time()
tasks = []
async for president in await iterate_presidents():
# Use asyncio.create_task to schedule the coroutine for execution before awaiting it
# This way descriptions will start being generated while the list of presidents is still being generated
task = asyncio.create_task(tell_me_more_about(president))
tasks.append(task)
descriptions = await asyncio.gather(*tasks)
Note: This was inspired by a Magentic example.
Currently validator.check_type does not support enum outputs types. If LLM outputs a valid integer, check_type outputs False where it should put True
class RequestIntent(Enum):
REFUNDS = 1
PRODUCT_QUESTIONS = 2
ACCOUNT_OPENING = 3
ORDER_STATUS = 4
ORDER_MODIFICATIONS = 5
@Monkey.patch
def classify_request(input: str) -> RequestIntent: # Multiclass classification
"""
Classify the incoming user request intent
"""
In many cases, writing to the disk is not possible or advised. There should be the option of other persistence layers that enable writing of data to S3 / Redis / Cloudflare KV etc. In order for this to be possible, a clean persistence interface must be created.
Describe the bug
When using a Bedrock teacher model, the finetuning should be disabled for the time being. However it is disabled wrongly.
To Reproduce
When using a Bedrock model, after 200 datapoints, tanuki will try to start a finetune job which results in a error
Expected behavior
When using Bedrock models, finetuning should not be used.
Input: player name
Output: List of greatest accomplishments by that player in the NBA
Something is going wrong with the state management.
When we run tests all together, several tests fail with errors such as:
self = <unittest.mock._patch object at 0x119b38650>
def get_original(self):
target = self.getter()
name = self.attribute
original = DEFAULT
local = False
try:
original = target.__dict__[name]
except (AttributeError, KeyError):
original = getattr(target, name, DEFAULT)
else:
local = True
if name in _builtins and isinstance(target, ModuleType):
self.create = True
if not self.create and original is DEFAULT:
> raise AttributeError(
"%s does not have the attribute %r" % (target, name)
)
E AttributeError: <module 'test_finance' from '/Users/.../PycharmProjects/monkeyFunctions/tests/test_patch/test_finance.py'> does not have the attribute 'classify_sentiment_2'
This is likely due to how mocked functions are scoped.
Input: 1) notes, 2) intended recipient name, 3) type of email, 3) short/medium/long
Output: email draft
Sample notes to use: https://docs.google.com/document/d/1gPGJu6bpDqTb1X5ye63uWZ4jPIp4AQbSRhfHjdV-OcM/edit#heading=h.1gqm9d6ikfhc
Is your feature request related to a problem? Please describe.
Running any function is expensive due to the network hop required.
Describe the solution you'd like
Automatic caching to memory, or to a designated provider (e.g Redis)
Describe alternatives you've considered
Asking the user to use an existing function caching library such as @functools.cache
Hi,
I'd like to join the Discord group but the link invite is listed as invalid.
Any way a fresh link can be provided?
Input: 1) list of ingredients available, 2) describe what cocktails you like, 3) how many recommendations you'd like
Output: Generates X many recommendations of cocktails with % match of ingredients to what you have available
Use this API: https://www.thecocktaildb.com/api.php
Input: Query (eg "fine-tuning LLMs"), minimum number of comments, how frequently to check per day, email)
Output: Emails the link to the post + summary after each run
Sources: https://hn.algolia.com/api, https://modal.com/docs/examples/hackernews_alerts
The following align fails, where the float is negative. If the float is positive (0.5), the align does not fail. The same happens with ints, negative ints raise an error, positive ints do not
class ArticleSummary(BaseModel):
sentiment: float = Field(..., ge=-10, le=1.0)
@tanuki.align
def align_analyze_article():
html_content = "<head></head><body><p>Nvidia has made the terrible decision to buy ARM for $40b on 8th November. This promises to "\
"be an extremely important decision for the industry, even though it creates a monopoly.</p></body> "
assert analyze_article_3(html_content, "nvidia") == ArticleSummary(
sentiment=-0.5,
)
align_analyze_article()
File "site-packages\tanuki\assertion_visitor.py", line 424, in extract_output
raise NotImplementedError(f"Node type {type(node).__name__} not handled yet")
NotImplementedError: Node type UnaryOp not handled yet
Replacing regex for email-to-name matching
Create support for pydantic objects as inputs to align statements and patched functions
The following tests aren't passing
test_assertion_visitor/test_mock -- test_rl_equality
test_validator/test_instantiate -- test_generics, test_extended_generics
I am using the exact code provided in an example and I have an existing openAI API key that I have tested in another script calling openAI directly.
Please help me understand why it's providing a generic exception response "Exception: OpenAI API failed to generate a response"
`@monkey.patch
def score_sentiment(input: str) -> Annotated[int, Field(gt=0, lt=10)]:
"""Scores the input between 0-10"""
@monkey.align
def align_score_sentiment():
"""Register several examples to align your function"""
assert score_sentiment("I love you") == 10
assert score_sentiment("I hate you") == 0
assert score_sentiment("You're okay I guess") == 5
def test_score_sentiment():
"""We can test the function as normal using Pytest or Unittest"""
score = score_sentiment("I like you")
assert score >= 7
if name == "main":
align_score_sentiment()
print(score_sentiment("I like you")) # 7
print(score_sentiment("Apples might be red")) # `
Traceback (most recent call last): File "/Users/jacoblaws/Development/python/eleva/gpt_experiments/monkey_patch_descriptions.py", line 46, in <module> print(score_sentiment("I like you")) # 7 File "/Users/jacoblaws/opt/anaconda3/lib/python3.9/site-packages/monkey_patch/monkey.py", line 206, in wrapper output = Monkey.language_modeler.generate(args, kwargs, Monkey.function_modeler, function_description) File "/Users/jacoblaws/opt/anaconda3/lib/python3.9/site-packages/monkey_patch/language_models/language_modeler.py", line 31, in generate choice = self.synthesise_answer(prompt, model, model_type, llm_parameters) File "/Users/jacoblaws/opt/anaconda3/lib/python3.9/site-packages/monkey_patch/language_models/language_modeler.py", line 41, in synthesise_answer return self.api_models[model_type].generate(model, self.system_message, prompt, **llm_parameters) File "/Users/jacoblaws/opt/anaconda3/lib/python3.9/site-packages/monkey_patch/language_models/openai_api.py", line 70, in generate raise Exception("OpenAI API failed to generate a response") Exception: OpenAI API failed to generate a response
The following align statements error out with inputs Lists or tuples
@Monkey.patch
def classify_sentiment(input: List[str]) -> Literal['Good', 'Bad', 'Neutral']: # Multi-class classification
"""
Determine if the input is positive, negative or neutral sentiment
"""
@Monkey.align
def align():
assert classify_sentiment(["I thought the ending was awesome"]) == 'Good'
assert classify_sentiment(["The acting was horrendous"]) == 'Bad'
assert classify_sentiment(["It was a dark and stormy night"]) == 'Neutral'
@Monkey.patch
def classify_sentiment(input: tuple) -> Literal['Good', 'Bad', 'Neutral']: # Multi-class classification
"""
Determine if the input is positive, negative or neutral sentiment
"""
@Monkey.align
def align():
assert classify_sentiment(("I thought the ending was awesome", "It was really good")) == 'Good'
Currently other than if the key is invalid, if the retry mechanism fails for OpenAI generation, the error returned is a generic error message. We should return the specific error message that OpenAI return on the final retry
Demonstrate how to build an app using Vercel with FastAPI+MonkeyPatch deployed onto Vercel serverless.
Make sure to document your process and write some written content to accompany it.
Hey! This is a super cool project.
I noticed your comment on https://news.ycombinator.com/item?id=38285191 about supporting open source models. I'm from the Anyscale team and we offer an OpenAI compatible API for both fine-tuning and serving for the most popular Open Source models (Llama-2, Codellama, Mistral, Zephyr).
Perhaps this would be a fast way to support open source models for monkeypath.py?
Currently, we automatically track usage of MP to help understand how people are using it.
There is currently no way to turn this off. Users may want to disable telemetry for privacy reasons.
We should add a top-level global config parameter for users to opt-out if necessary. The syntax could look something like this:
Monkey.telemetry(disable=True)
Pydantic fields include constraints on the content, e.g a character limit:
from pydantic import BaseModel, Field
from typing import Optional
class MyPydanticModel(BaseModel):
title: Optional[str] = Field(None, max_length=10)
We should support this, and all other Field contraints.
Patched Monkey functions should support operating in 'encoder-only' mode, in which a function returns a vector that can be used for indexing and retrieving from a VectorDB.
In this issue, I propose 2 syntactic options for supporting vector responses.
A new decorator: embed
that informs the library that we want to return a vector, rather than a decoded type. By passing in an array-like type into the decorator, we can specify the type we want to construct, i.e np.array
or list[float]
.
Example:
@monkey.embed(as=np.array)
def score_sentiment(input: str) -> Annotated[int, Field(gt=0, lt=10)]:
"""
Scores the input between 0-10
"""
This returns an Embedding[np.array]
when called.
Pro:
Con:
Example:
@monkey.patch
def score_sentiment(input: str) -> Embedding[np.array[float]]
"""
Scores the input between 0-10
"""
This returns an Embedding[np.array]
when called.
Pro:
Con:
I order for us to be able to align the embeddings to our use-case, we need syntax that supports assert
as well as the Embedding object.
@monkey.align
def align_score_sentiment():
"""Register several examples to align the embedding function"""
assert score_sentiment("I love you") == score_sentiment("I really love you")
assert score_sentiment("I hate you") != score_sentiment("I love you")
We can possibly consider custom methods on the Embedding object, i.e Embedding.similar
, but this may not be necessary.
In order to fine-tune our aligned embedding functions on the backend, we can use constrastive finetuning - but this is not possible using the OpenAI API.
We are planning to support returning embeddings from OpenAI. We should generalize this to support embeddings from Bedrock (and any other LLM providers that we choose to support).
Is your feature request related to a problem? Please describe.
LLMs are extremely inefficient at classification. XGBoost is better if the data is available. We could use the aligned data from the LLM to train an XGBoost model, which would be much faster to run.
Describe the solution you'd like
When the output types denote a classification task (i.e where the goal is to sample one type in a union of literal types, or an enum), we optionally distil the teacher model into a decision forest using the XGBoost library.
Additional context
We could represent student models as optional packages, sort of like drivers, that the user could install through PIP.
E.g pip3 install tanuki.py[xgboost]
Is your feature request related to a problem? Please describe.
Too many anonymous telemetry calls are being sent. This creates unnecessary overhead.
Describe the solution you'd like
Reduce the number of calls made, and make each one more informative while still being anonymized.
E.g:
If the output class definition is a Pydantic class, the prompt sent to LLM includes the definition of that class. If it's a Optional or Union, the prompt does not include that
During application startup for the todolist example (examples/todolist/backend/main.py) the following error occurs
Error
monkey-patch\src\assertion_visitor.py", line 254, in extract_output raise NotImplementedError(f"Function {func_name} not handled yet") NotImplementedError: Function TodoItem not handled yet
Some non-fatal errors use print (i.e 'Could not add datapoint to training data'). We should use the logger.
Currently there is no automatic way to ensure that tests pass before code is merged into the trunk. This should be a priority.
OSError Traceback (most recent call last)
Cell In[67], line 1
----> 1 x = create_todolist_item("I need to go and visit Jeff at 3pm tomorrow")
2 print(x)
File ~/Paperplane/repos/monkey-patch.py/examples/wikipedia/../../src/monkey.py:202, in Monkey.patch..wrapper(*args, **kwargs)
200 @wraps(test_func)
201 def wrapper(*args, **kwargs):
--> 202 function_description = Register.load_function_description(test_func)
203 f = str(function_description.dict.repr() + "\n")
204 output = Monkey.language_modeler.generate(args, kwargs, Monkey.function_modeler, function_description)
File ~/Paperplane/repos/monkey-patch.py/examples/wikipedia/../../src/register.py:86, in Register.load_function_description(func_object)
80 # Extract class definitions for input and output types
81 input_class_definitions = {
82 param_name: get_class_definition(param_type)
83 for param_name, param_type in input_type_hints.items()
84 }
---> 86 output_class_definition = get_class_definition(output_type_hint)
88 return FunctionDescription(
89 name=func_object.name,
90 docstring=docstring,
(...)
94 output_class_definition=output_class_definition
95 )
File ~/Paperplane/repos/monkey-patch.py/examples/wikipedia/../../src/register.py:77, in Register.load_function_description..get_class_definition(class_type)
75 return [get_class_definition(arg) for arg in class_type.args if arg is not None]
76 elif inspect.isclass(class_type) and class_type.module != "builtins":
---> 77 return inspect.getsource(class_type)
78 return class_type.name
File /usr/local/Cellar/[email protected]/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/inspect.py:1262, in getsource(object)
1256 def getsource(object):
1257 """Return the text of the source code for an object.
1258
1259 The argument may be a module, class, method, function, traceback, frame,
1260 or code object. The source code is returned as a single string. An
1261 OSError is raised if the source code cannot be retrieved."""
-> 1262 lines, lnum = getsourcelines(object)
1263 return ''.join(lines)
File /usr/local/Cellar/[email protected]/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/inspect.py:1244, in getsourcelines(object)
1236 """Return a list of source lines and starting line number for an object.
1237
1238 The argument may be a module, class, method, function, traceback, frame,
(...)
1241 original source file the first line of code was found. An OSError is
1242 raised if the source code cannot be retrieved."""
1243 object = unwrap(object)
-> 1244 lines, lnum = findsource(object)
1246 if istraceback(object):
1247 object = object.tb_frame
File /usr/local/Cellar/[email protected]/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/inspect.py:1063, in findsource(object)
1055 def findsource(object):
1056 """Return the entire source file and starting line number for an object.
1057
1058 The argument may be a module, class, method, function, traceback, frame,
1059 or code object. The source code is returned as a list of all the lines
1060 in the file and the line number indexes a line in that list. An OSError
1061 is raised if the source code cannot be retrieved."""
-> 1063 file = getsourcefile(object)
1064 if file:
1065 # Invalidate cache if needed.
1066 linecache.checkcache(file)
File /usr/local/Cellar/[email protected]/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/inspect.py:940, in getsourcefile(object)
936 def getsourcefile(object):
937 """Return the filename that can be used to locate an object's source.
938 Return None if no way can be identified to get the source.
939 """
--> 940 filename = getfile(object)
941 all_bytecode_suffixes = importlib.machinery.DEBUG_BYTECODE_SUFFIXES[:]
942 all_bytecode_suffixes += importlib.machinery.OPTIMIZED_BYTECODE_SUFFIXES[:]
File /usr/local/Cellar/[email protected]/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/inspect.py:908, in getfile(object)
906 return module.file
907 if object.module == 'main':
--> 908 raise OSError('source code not available')
909 raise TypeError('{!r} is a built-in class'.format(object))
910 if ismethod(object):
OSError: source code not available
Describe the bug
When getting OpenAI finetunes, the FinetuneJob
is instantiated with a string as fine_tuned_model
not a BaseModelConfig
To Reproduce
Steps to reproduce the behavior:
Start a OpenAI finetuning job and see the FinetuneJob
Expected behavior
FinetuneJob
should have a BaseModelConfig
as the fine_tuned_model
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Smartphone (please complete the following information):
Additional context
Add any other context about the problem here.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.