Comments (10)
ok. then yes.
To summarize why i think this makes sense:
An endpoint could be made public to people who didn't create the endpoint and thus the path name should match the functionality to reduce confusion.
in this case it makes sense to enable something similar to what @aniketmaurya proposed in the beginning here
# default
LitServer(..., api_path="predict")
# overwrite it (for example to be called /classify)
LitServer(..., api_path="classify")
but the method name still remains "predict" in the server API.
wdyt @aniketmaurya @lantiga
from litserve.
hi @eugsokolov, thanks for all the feedback. For manually specifying endpoint, the following looks ideal to me.
# default
LitServer(..., api_path="predict")
# overwrite it (for example to be called /classify)
LitServer(..., api_path="classify")
@eugsokolov for multiple endpoints, I would suggest to use LitSpec
(note that it is not 100% documented yet), we use it to provide OpenAI compatible API endpoints here.
from litserve.
yeah not sure about multiple endpoints on the same server. because now you need a different encode and decode function for each as well.
i don’t like building “nice to haves” for “theoretical scenarios”. do you have a concrete, real-world example where you must have multiple methods?
i totally get it for regular web development… but i don’t see how that applies here and i’m not even sure we would even want to encourage such a thing. i can see that being fraught with problems like out of memory, collisions, etc….
again, i don’t want to jump to an answer here, we need to understand the problem yet. i understand in “theory” it would be “nice” to do X… but how realistic is this? products get overbloated with “nice” things that could happen in “theory” and we know 99.999% of the time it never comes up.
from litserve.
@aniketmaurya @williamFalcon hi
i am curious what thoughts are around standardization. it would be nice to change the default /predict
or to add new endpoints (doesnt seem to be possible now but I could be wrong). having something like /classify
or /cluster
or the like might be interesting for some users. having flexibility to change the default (via decorator) might be nice
from litserve.
hi @eugsokolov, may I know more about your use case, please? What can /classify
solve that /predict
is not able to? Some context will be helpful.
from litserve.
hi @eugsokolov, may I know more about your use case, please? What can
/classify
solve that/predict
is not able to? Some context will be helpful.
/predict
can do everything /classify
can do, but sometimes a model may not want to provide a prediction but instead do something else. there might be a model that can predict and classify (and do something else) as well and it might be nice to have multiple endpoints all under one LitAPI
to clarify - i think this would be a nice to have and was curious what the rationale in the PR around "this is breaking our standardization" were
from litserve.
i’m super interested in understanding what problem this solves so we can help. just need more details 😉
the word predict is being taken too literally here. predict just means “do what you want now with your model”. if you want to classify, segment, summarize, etc… just do it.
it’s just a word.
so, i’m still having a hard time understanding what the problem is that you are trying to solve?
“with predict i cannot do XYZ and thus we need ABC to enable the behavior i am looking for”.
without a clear motivation, this feels purely aesthetic…
are you asking that the endpoint /predict be customizable?
are you asking for the literal “predict” method to be named something else? if this is the case, why does the name matter?
are you asking for a server to provide multiple functionalities? ie: predict AND classify? and if so, is that really a best practice? do you have a concrete example where this is used in a real world application (not just “nice” in theory?) and if that’s further true, can’t a simple if statement handle that logic for you in the predict step?
we aim to provide a standard that allows servers to interoperate and be legible by anyone. if every litserve implementation uses a different name it loses the ability to be standardized… ie: right now if you want to know what a server is doing, then simply look at the predict method. if we remove this standard, it will be hard to pinpoint exactly what a server is doing to someone not familiar with a code base.
not saying no to the request, just asking to understand the underlying problem to understand what a reasonable solution might be.
from litserve.
yes this is aesthetic - if i am exposing an endpoint to the public i may not want it to be called /predict
if it isnt providing any sort of prediction. i have a model that is classifying geospatial data and not doing any predictions but i am forced to use the /predict
endpoint (openai doesnt even have a predict endpoint, just completion and embedding). in similar vein, i name my variables to something that resembles the data they are storing.
i dont feel very strongly here and thought it would be nice to name the endpoint via a decorator instead of always needing to use the default name
as far as best practice to provide multiple endpoints, maybe i want to provide a debug or data endpoint for my users. if there isnt much flexibility here then ill create a fastapi app to provide the relevant endpoints for my users
from litserve.
ok. then yes.
To summarize why i think this makes sense:
An endpoint could be made public to people who didn't create the endpoint and thus the path name should match the functionality to reduce confusion.
in this case it makes sense to enable something similar to what @aniketmaurya proposed in the beginning here
# default LitServer(..., api_path="predict") # overwrite it (for example to be called /classify) LitServer(..., api_path="classify")but the method name still remains "predict" in the server API.
wdyt @aniketmaurya @lantiga
awesome!
i'll also add that a partial requirement of "renaming" would be to add new endpoints (which of course would require a different name) and hence it may make sense to force the developer to name the endpoint. this seems like a paradigm shift from the current implementation. but i do believe that taking an example from fastapi would be useful here in allowing for developers the flexibility.
the default example can include /predict
. what i am envisioning would look something like:
# server.py
import litserve as ls
# STEP 1: DEFINE YOUR MODEL API
class SimpleLitAPI(ls.LitAPI):
def setup(self, device):
# Setup the model so it can be called in `predict`
self.model = lambda x: x**2
def decode_request(self, request):
# Convert the request payload to your model input.
return request["input"]
@ls.endpoint("/predict")
def predict(self, x):
# Run the model on the input and return the output.
return self.model(x)
@ls.endpoint("/classify")
def classify(self, x):
return self.model.classify(x)
@ls.endpoint("/model-weights")
def model_weights(self, x):
return self.model.weights
def encode_response(self, output):
# Convert the model output to a response payload.
return {"output": output}
# STEP 2: START THE SERVER
if __name__ == "__main__":
api = SimpleLitAPI()
server = ls.LitServer(api, accelerator="auto")
server.run(port=8000)
from litserve.
Interestingly, we already have the concept of endpoint
with their own path
(and method
) behind the scenes, which we currently use to expose /v1/chat/completion
for the OpenAI spec.
The only question is, do we really need to expose it to users? Flexibility comes with cognitive overhead so we need to be super conservative. In the future, as practical needs arise, we can expose the mechanism for advanced users, but probably not before we have a real need.
For now @aniketmaurya 's solution seems to check all boxes so it's a +1 for me.
from litserve.
Related Issues (20)
- failed to start and serve HTTP while server is in intermediate state i.e. shutting down HOT 5
- Feature: Add support to include tools for OpenAI Chat Template to enable function calling. HOT 3
- Question: Serving LLama3-70B on multiple GPUs HOT 4
- during manual local testing, the processes are not killed if the test fails
- Track device automatically.
- OpenAI spec non-streaming response contains multiple `space` tokens HOT 2
- document better HOT 3
- CI issues HOT 3
- propagate Exception from inference workers to main process
- Allow LitAPI to raise FastAPI `HTTPException`
- API monitoring metrics HOT 2
- Evict requests if the client has disconnected HOT 4
- Map `decode_request` during dynamic batching using a threadpool
- Prevent large payload attack
- Feature request: Docker image for LitServe HOT 1
- Fix test_timeout test flake
- Show warning if `batch` & `unbatch` is implemented but max_batch_size not set in `LitServer` HOT 2
- Add health check endpoint HOT 4
- move `wrap_litserve_start` to utils HOT 5
- Add support for `response_format` in OpenAI Spec HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from litserve.