ch3njust1n / smart Goto Github PK

https://rebuff.ai/

https://github.com/woop/rebuff

1. pip install mypy
2. Add to setup.py dev section
3. Update pre-commit-config.yaml
4. Update github workflow dynamic.yaml

_is_generative property in decorators and metaclasses indicate a function or class has access to a generative model.
Currently unclear how this could be used, but it gives the rest of the program information on what is generative and what is now.

Here is an example that could be used so that a LLM knows it can also manipulate other parts of the program:

import inspect
import textwrap
from meta import gpt4

all_methods = inspect.getmembers(
    cls, predicate=inspect.isfunction
)
available_funcs = [
    (
        method[0],
        textwrap.dedent(inspect.getsource(method[1])),
    )
    for method in all_methods if method._is_generative
]

func_source = gpt4(f'Generate code with these generative functions: {available_funcs}')

• Add unit tests
• Documentation

Definition: This involves changing or extending the behavior of code at runtime, typically by adding, replacing, or modifying methods or attributes in a class or object.

Example:

class MyClass:
    pass

def new_method(self):
    print("New method")

MyClass.new_method = new_method
instance = MyClass()
instance.new_method() # Prints: "New method"

Task:

• LLM generates the new_method and returns a pointer to the function from exec() and assigns to MyClass.new_method
• Add example to documentation

https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-citation-files

• Store generated functions for later use and versioning
• Add example to documentation

• Generate functions in response to an action for a function that does not exist
• Unit test

Definition: You can create classes dynamically using the type function.

Example:

def my_method(self):
    return self.x * 2

MyClass = type('MyClass', (object,), {'x': 10, 'my_method': my_method})
instance = MyClass()
print(instance.x)  # Output: 10
print(instance.my_method())  # Output: 20

Task:

• LLM generates the parameters given to type()
• Add example to documentation

Allow user to provide their own callback to decorator so they can specify their own LLM solution

Definition: These are similar to function decorators, but they modify or extend classes instead.

Example:

def my_class_decorator(cls):
    cls.decorated = True
    return cls

@my_class_decorator
class MyClass:
    pass

print(MyClass.decorated)  # Output: True

instance = MyClass()
print(instance.decorated)  # Output: True

Task:

• LLM generates the decorators properties and values given the current class definition. Similar to function decorators.
• Add example to documentation

Use python's ast package to check the syntax of generated code at runtime.

import ast

def is_code_correct(code):
    try:
        ast.parse(code)
        return True
    except SyntaxError:
        return False

# test
print(is_code_correct('print("Hello, world!")'))  # True
print(is_code_correct('print("Hello, world!'))  # False - missing closing quote

• Unit tests
• Documentation

Factory function that given a data structure, it returns a new generative data structure
e.g.

from collections import deque
from generative.structures import convert

q = convert(deque)

Under the hood it could be implemented as follows:

import builtins
from generative import adapt

vars().update({k: adapt.adapt(v) if callable(v) else v for k, v in vars(builtins).items() if not k.startswith("__")})

Import all python modules:

import os
import importlib
import sys

# Get the path of Python's lib directory
lib_path = os.path.dirname(os.__file__)

# Get a list of all .py files in lib directory
module_files = [f for f in os.listdir(lib_path) if f.endswith('.py')]

# Remove the .py extension to get the module names
module_names = [f[:-3] for f in module_files]

# Import the modules
native_modules = []
for name in module_names:
    try:
        module = importlib.import_module(name)
        native_modules.append(module)
    except:
        pass  # If a module fails to import, just skip it

print(native_modules)  # This prints the list of imported native modules

Then can modify all native functionality with generative.adapt(<place functionality here>), but would need to modify the comprehension so that it recursively applies @adapt to functions of objects or use metaclasses.generate().

https://securityscorecards.dev/

• Make suggestions for improving code and combine with existing profiling tools
• Add example to documentation

Task:

• Integrate Claude API so can compare with GPT-4.
• Document example usage

Add color to stack_trace so it's clear what part was generated

Tasks:

• Integrate Cohere LLM
• Document example usage

Generate a large unit test/ benchmark dataset for paper. Tests should be random and range in complexity.

Definition: A metaclass is the class of a class, which Python uses to create and control classes just like classes create and control objects. The default metaclass is type, but you can create your own metaclass by inheriting from type.

Example:

class MyMeta(type):
    def __init__(cls, name, bases, dct):
        cls.class_created_by = "MyMeta"
        super().__init__(name, bases, dct)

class MyClass(metaclass=MyMeta):
    pass

print(MyClass.class_created_by)  # Output: MyMeta

Task:

• LLM generates metaclass properties
• Add example to documentation

Translate to Mojo

1. What is code injection
2. Types of code injection and examples
3. How to execute a code injection
4. Code injection with respect to metaprogramming
5. How to prevent code injections
6. How we're using code injection
7. Citations (find papers on machine learning and cybersecurity or related papers)

1. Attacks against our approach
2. Defenses against our approach

Add a table with

• Decorators
• Metaclasses
• Dynamic class generation

Example from Torch-Struct: Deep Structured Prediction Library paper

Example: https://docs.lmql.ai/en/stable/

Use GPT-4 to generate code and a faster model with more context like claude-instant-v1-100k.
User should be able to define their own semantic checking model and the metaprogramming API should optionally expect a semantic checker as input along with the generating model (model input parameter).

• Unit test
• Documentation

Dynamic Metaprogramming with Large Language Models

Sections:

1. Abstract

2. Introduction

• Metaprogramming
  -- What is metaprogramming
  -- Types of metaprogramming
  -- Why use metaprogramming

• Code injection
  -- What is code injection
  -- What are the different types of code injection
  -- How to mitigate most common attacks
  -- How does this relate to metaprogramming

3. Related work

4. Dynamic Metaprogramming with Large Language Models

5. Results

6. Considerations

• Practicality
  -- Latency
  -- Compute & Cost
• Secure code
• Best practices

7. Risks for run-time metaprogramming

• Difficulty in directing outputs of LLM
• Difficulty of guaranteeing correctness of LLM outputs

8. References

• Capture stack trace and summarize and make suggestions.
• Add example to documentation

• Log LLM generated functions