UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.
Home Page: https://uptrain.ai/
License: Apache License 2.0
Japan is a major player in the technology industry, and many developers in Japan use Japanese as their primary language for development. It would be great to have the README.md translated into Japanese so that more developers can quickly understand what UpTrain tool does.
Currently, we have only one measure of recommendation bias in the repository: Popularity Bias.
It would be great to have many other measures of bias implemented in the repository, such as exposure bias, position bias, bias towards any particular feature (such as race and gender), etc.
You can check out this blog to learn more about biases in recommendation models.
Improve the designs for the streamlit dashboard.
We currently support only custom-defined metrics for edge cases and data integrity.
Add boxplots and Tukey's fences for edge case checks in the UpTrain frameworks to identify outliers in data.
A translation for README.md in Chinese would allow more developers to understand what UpTrain is about quickly.
The package currently saves a copy of everything it sees on disk. It is unnecessary. Reduce logging to minimal while making sure that the tests still pass.
We currently use Streamlit for dashboarding and have a custom data-logging functionality that logs the data into CSV files to be read by streamlit.
UpTrain config has parameters (st_logging) to turn streamlit logging on and off (https://github.com/uptrain-ai/uptrain/blob/main/uptrain/core/classes/helpers/config_handler.py)
Depending upon the config, the Log Handler class (https://github.com/uptrain-ai/uptrain/blob/main/uptrain/core/classes/logging/log_handler.py) - has functions like "add_scalars", "add_histogram", "add_alert", etc. which takes in the data and plot them onto dashboards.
The integration will be primarily done in the Log Handler class
Grafana - https://github.com/grafana/grafana
Prometheus - https://github.com/prometheus/prometheus
The function attach_ground_truth() in framework.py is slow. For the concept drift example, it takes ~10x the time of model prediction. It can be optimized further.
Add an ML model monitoring example in the Reinforcement Learning domain.
Spanish is widely spoken in many countries, including Spain, Mexico, and South America. Many software developers and companies in these regions use Spanish as their primary language for development. It would be great to have a version of README.md in Spanish for these developers to get started with UpTrain quickly.
Allow defining custom visualizations to explore and understand logged data, where users can define their own visualizations using libraries such as matplotlib or plotly.
Check: human_orientation_classification/deepdive_examples/uptrain_data_drift_custom_measures.ipynb
currently, we use bucketing and earth moving distance (checkout current implementation here). Can we implement better algorithms for calculating embedding drift that takes fewer hyperparameters and is more general?
Checkout out the file: https://github.com/uptrain-ai/uptrain/blob/main/uptrain/tests/test_dashboard.py
Here, streamlit dashboard runs in a separate thread and plots data that is logged in the folder uptrain_logs. Can we make the main thread and the streamlit thread communicate? This article might help: https://www.geeksforgeeks.org/python-communicating-between-threads-set-1/
The task is to create an illustration of the threads communicating. For example, we input something from the streamlit dashboard and it prints at the python console.
Histogram is not supported for streamlit
Originally posted by @sourabhagr in #50 (comment)
Remove Dependency on RecList in future commits
Originally posted by @sourabhagr in #19 (review)
Google colab does not seem to support multithreading (for instance, the thread for streamlit dashboards doesn't show up on Colab).
A translation for README.md in Russian would allow developers who prefer reading in Russian to understand what UpTrain is about quickly.
A translation for README.md in German would allow developers who prefer reading in German to understand what UpTrain is about quickly.
Step 4: Initialize the UpTrain Framework of the get_started.ipynb gives the following error on running.
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[14], line 17
1 cfg = {
2 "checks": checks,
3 "training_args": training_args,
(...)
13 "cluster_visualize_func": plot_all_cluster,
14 }
16 # Initialize the UpTrain framework object with config
---> 17 framework = uptrain.Framework(cfg)
18 print("Successfully Initialized UpTrain Framework")
File ~\anaconda3\lib\site-packages\uptrain\core\classes\framework.py:83, in Framework.__init__(self, cfg_dict)
81 self.model_handler = ModelHandler()
82 self.log_handler = LogHandler(framework=self, cfg=cfg)
---> 83 self.anomaly_manager = AnomalyManager(self, cfg.checks)
84 self.reset_retraining()
86 if training_args.data_transformation_func:
File ~\anaconda3\lib\site-packages\uptrain\core\classes\anomalies\managers\anomaly_manager.py:23, in AnomalyManager.__init__(self, framework, checks)
21 self.fw = framework
22 for check in checks:
---> 23 self.add_anomaly_to_monitor(check)
File ~\anaconda3\lib\site-packages\uptrain\core\classes\anomalies\managers\anomaly_manager.py:35, in AnomalyManager.add_anomaly_to_monitor(self, check)
32 elif check["type"] == Anomaly.DATA_DRIFT:
33 if "measurable_args" in check:
34 drift_managers = [
---> 35 DataDrift(
36 self.fw, check, is_embedding=check.get("is_embedding", None)
37 )
38 ]
39 else:
40 drift_managers = []
File ~\anaconda3\lib\site-packages\uptrain\core\classes\anomalies\data_drift.py:30, in DataDrift.__init__(self, fw, check, is_embedding)
28 self.count = 0
29 self.prod_dist_counts_arr = []
---> 30 self.bucket_reference_dataset()
File ~\anaconda3\lib\site-packages\uptrain\core\classes\anomalies\data_drift.py:284, in DataDrift.bucket_reference_dataset(self)
281 all_inputs = np.reshape(all_inputs, all_inputs_shape)
283 if self.is_embedding:
--> 284 self.bucket_vector(all_inputs)
285 else:
286 buckets = []
File ~\anaconda3\lib\site-packages\uptrain\core\classes\anomalies\data_drift.py:336, in DataDrift.bucket_vector(self, data)
334 def bucket_vector(self, data):
--> 336 all_clusters, counts, cluster_vars = cluster_and_plot_data(
337 data,
338 self.NUM_BUCKETS,
339 cluster_plot_func=self.cluster_plot_func,
340 plot_save_name="training_dataset_clusters.png",
341 )
343 self.clusters = np.array([all_clusters])
344 self.cluster_vars = np.array([cluster_vars])
File ~\anaconda3\lib\site-packages\uptrain\core\lib\helper_funcs.py:16, in cluster_and_plot_data(data, num_clusters, cluster_plot_func, plot_save_name)
12 def cluster_and_plot_data(
13 data, num_clusters, cluster_plot_func=None, plot_save_name=""
14 ):
15 kmeans = KMeans(n_clusters=num_clusters, random_state=1, n_init=10)
---> 16 kmeans.fit(data)
17 all_clusters = kmeans.cluster_centers_
18 all_labels = kmeans.labels_
File ~\anaconda3\lib\site-packages\sklearn\cluster\_kmeans.py:1455, in KMeans.fit(self, X, y, sample_weight)
1453 else:
1454 kmeans_single = _kmeans_single_lloyd
-> 1455 self._check_mkl_vcomp(X, X.shape[0])
1457 best_inertia, best_labels = None, None
1459 for i in range(self._n_init):
1460 # Initialize centers
File ~\anaconda3\lib\site-packages\sklearn\cluster\_kmeans.py:911, in _BaseKMeans._check_mkl_vcomp(self, X, n_samples)
909 n_active_threads = int(np.ceil(n_samples / CHUNK_SIZE))
910 if n_active_threads < self._n_threads:
--> 911 modules = threadpool_info()
912 has_vcomp = "vcomp" in [module["prefix"] for module in modules]
913 has_mkl = ("mkl", "intel") in [
914 (module["internal_api"], module.get("threading_layer", None))
915 for module in modules
916 ]
File ~\anaconda3\lib\site-packages\sklearn\utils\fixes.py:150, in threadpool_info()
148 return controller.info()
149 else:
--> 150 return threadpoolctl.threadpool_info()
File ~\anaconda3\lib\site-packages\threadpoolctl.py:124, in threadpool_info()
107 @_format_docstring(USER_APIS=list(_ALL_USER_APIS),
108 INTERNAL_APIS=_ALL_INTERNAL_APIS)
109 def threadpool_info():
110 """Return the maximal number of threads for each detected library.
111
112 Return a list with all the supported modules that have been found. Each
(...)
122 In addition, each module may contain internal_api specific entries.
123 """
--> 124 return _ThreadpoolInfo(user_api=_ALL_USER_APIS).todicts()
File ~\anaconda3\lib\site-packages\threadpoolctl.py:340, in _ThreadpoolInfo.__init__(self, user_api, prefixes, modules)
337 self.user_api = [] if user_api is None else user_api
339 self.modules = []
--> 340 self._load_modules()
341 self._warn_if_incompatible_openmp()
342 else:
File ~\anaconda3\lib\site-packages\threadpoolctl.py:373, in _ThreadpoolInfo._load_modules(self)
371 self._find_modules_with_dyld()
372 elif sys.platform == "win32":
--> 373 self._find_modules_with_enum_process_module_ex()
374 else:
375 self._find_modules_with_dl_iterate_phdr()
File ~\anaconda3\lib\site-packages\threadpoolctl.py:485, in _ThreadpoolInfo._find_modules_with_enum_process_module_ex(self)
482 filepath = buf.value
484 # Store the module if it is supported and selected
--> 485 self._make_module_from_path(filepath)
486 finally:
487 kernel_32.CloseHandle(h_process)
File ~\anaconda3\lib\site-packages\threadpoolctl.py:515, in _ThreadpoolInfo._make_module_from_path(self, filepath)
513 if prefix in self.prefixes or user_api in self.user_api:
514 module_class = globals()[module_class]
--> 515 module = module_class(filepath, prefix, user_api, internal_api)
516 self.modules.append(module)
File ~\anaconda3\lib\site-packages\threadpoolctl.py:606, in _Module.__init__(self, filepath, prefix, user_api, internal_api)
604 self.internal_api = internal_api
605 self._dynlib = ctypes.CDLL(filepath, mode=_RTLD_NOLOAD)
--> 606 self.version = self.get_version()
607 self.num_threads = self.get_num_threads()
608 self._get_extra_info()
File ~\anaconda3\lib\site-packages\threadpoolctl.py:646, in _OpenBLASModule.get_version(self)
643 get_config = getattr(self._dynlib, "openblas_get_config",
644 lambda: None)
645 get_config.restype = ctypes.c_char_p
--> 646 config = get_config().split()
647 if config[0] == b"OpenBLAS":
648 return config[1].decode("utf-8")
AttributeError: 'NoneType' object has no attribute 'split'
- `custom initialize` and `custom check` functions are mutating attributes on a foreign object, which is not great. Maybe use a generator as an interface, since you are performing a `reduction` over a stream of input data (logs). Generators might feel uncomfortable for many users though.
def custom_metric():
initial_acc = None
acc_arr = []
count = 0
thres = 0.02
window_size = 200
is_drift_detected = False
while True:
inputs, outputs, gts, extra_args = (yield)
count += 1
acc_arr.append(outputs[0]==gts[0])
...
# call it as (might be incorrect, I don't remember passing data to a generator repeatedly)
for _ in custom_metric():
...Edit: I remember this as a neat talk on generators as data interfaces, definitely worth going through. Link - http://www.dabeaz.com/generators/index.html
Originally posted by @ananis25 in #13 (comment)
One interesting blog on this: https://mindfulmodeler.substack.com/p/shap-is-not-all-you-need
t-SNE visual in UpTrain currently supports only default hyperparameters. Make it general so that any parameter for t-SNE from sklearn (such as n_components, perplexity, n_iter) can be supported through the config.
Recommendation bias implies that bias monitoring happens only for recommender systems. However, bias can be present in any model that makes predictions (for example, a model shortlisting applicants but prefers CVs of white males, implying racism and gender bias).
We want to change all mentions of recommendation bias, starting with this file, to model bias.
This is also a good first issue to get started with UpTrain.
"Monitor" is a better name for the Anomaly classes currently defined in the UpTrain framework. We want to replace the words Anomaly (Anomalies) with Monitor (Monitors), respectively.
This is a good first issue for users to get acquainted with the UpTrain framework.
No need to plot both - we can plot the one which the user specified
We currently support only custom-defined metrics for edge cases and data integrity.
Add Z-score analysis in UpTrain to identify outliers in the data.
More info here - https://uptrain.ai/blog/bias-in-recommendation-systems
cfg should also have a defined schema, where you specify what properties are legal and can be used. Could use the pydantic library to define those.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.