vktrbr / optimization_ml

Optimization in ML

Hi

optimization_ml's Issues

Regression documentation

Make regression documentation

References: multidimensionaloptimization.pdf

• Conceptual model rst file ConceptualModel.rst. Describe the situation in which we work, e.g. stock exange or healthcare

• Mathematical model rst file MathematicalModel.rst. Describe losses and motivation or l1 and l2.
  (L2 - https://en.wikipedia.org/wiki/Tikhonov_regularization,
  L1 - https://en.wikipedia.org/wiki/Lasso_(statistics))

• Requirements rst file SolutionRequirements.rst. A very short part. About half a sheet

  1. Input
  2. visualiztion
  3. method

• Sphinx deploy sphinx-step-by-step.txt

• Formatting all files for latex. And create documentation in pdf format

MultiDimensional visualization

Make visualization

for 1-variable functions

• animation plot with the movement of the point

for 2-variables functions

• make contour line plot
• add movement of the point
• add movement of the point on the contour plot

for all functions

• graph of the decreasing gradient norm and the function values for each iteration.

Create video maker for gss

Plotting functions and points frame by frame

• Points moving
• Static function
• x-axis, y-axis formatting
• title formatting
• integrate to streamlit

Refactoring common function

Problems

Some features, such as gradient, are common to different projects.

Requirements

Refactoring the project for a clear import.

Brent vis

gitlab

Parabola visualization

Create spi visualization function

Input

• function
• history
• bounds

Output

• go.Figure with frames per each step

We have three points for each iteration. Therefore, we can build a parabolic equation and draw a parabola for each iteration

Notebook refactoring

Example notebook

Make user guide by function from

• algorithms.py
• visualization.py

by refactoring
example_notebook.ipynb

Parsing

Preparation

Make pd.DataFrame from given excel or csv file

Input data format: a table is given, where the column headers are passed in the first row. Indexes are missing

• Chek file format
• Make a list with names of columns
• Make a DataFrame

Create a conceptual model

Documentation

Make documentation for Interior-point methods

• Conceptual model rst file. Describe the situation in which we work

• A fully mathematical analysis with latex/rst records

• Algorithms devs

• Other files:
  Input
  visualiztion

• Formatting all files for latex. And create documentation in pdf format

Create a parent class for the optimization methods

Writing comments to code

Add comments

For each file below need to add comments.

• gradient_descent_constant_step.py

• gradient_descent_frac_step.py

• gradient_descent_optimal_step.py

• support.py

• make requirements.txt files

Comments must be written by pep and be fully understandable

Deploy the application on the streamlite site

Page

Is subtask of Streamlit. 1d algorithms

Making the page

Notebooks

Make an examples

• with neural network, example of too many dimensional function
• with a suitable interesting function
• measure the execution time of each method

Algorithms

• Logistic Regression
• Logistic Regression with radial basis functions (RBF)
• LR wirth L1
• SVM

Development of the Golden-section search

Development of the Golden-section search

Algorithm

Problem:
f(x) -> *extremum* on [a, b]

1. Receivea, b, e (e - error/accuracy)
2. Calculate:
   x1 = b - (b - a) / Ф,
   x2 = a + (b - a) / Ф
3. if f(x1) > f(x2) (for min) | if f(x1) > f(x2) (for max) then a = x1
   else b = x2
4. Repeat 2, 3 steps while |a - b| > e

Ф = (1 + 5 ** 0.5) / 2

Parametres

Function will receive:

• Callable which depend on the first positional argument and other kwargs
• optimization bounds
• optimization accuracy
• type optimization (min, max)
• maximum number of iterations
• verbose. print iteration logs
• dataset output flag

Faction's output:

• extremum point
• success flag (0 - extremum found; 1 - max iteration; 2 - exception / error)

Vizualization

For one dimensional unputs make

• skatter plot
• line plot
• comparison plot with different regulators

Python Packages

Inputs (for each func)

• array of predictors (np.array) (add assert for type np.ndarray)
• array of the predicted variable (add add assert for type np.ndarray)
• type of regulators (l1, l2, none)

Output (for each func)

• function in analytical form (is it necessary?)
• weights of each x (yea)
• finding a free regression term (is it necessary?)

Functions

reference: https://scikit-learn.org/stable/modules/linear_model.html#ordinary-least-squares

Loss:

L1:

L2:

• Linear regression with L1
• Linear regression with L2
• Polynomial with L1

1. Make polynomial features with sklearn: PolynomialFeatures
2. Make linear regression on this features

• Polynomial with L2

exponential:

Let's make THETA = [THETA0, *THETA1] is m dimensional vector-column.
And try:

1. minimize sse loss without logarithmic
   
2. minimize after logarithmic

• Exponential with L1
• Exponential with L2

Notebook

For file 'why regression is so important'

• Add example of L1 regularization
• format the x-value(as in first example), legend graphic
• comments

For example_notebook

• make regression using all methods with stock market data. (RUBCNY for last 5 weeks)
• make comparison with plots.

Python Packages

Development of 3 methods

Made on GO

Possible inputs

• Function: Callable
• Limitations of the type of equalities and inequalities: Callable
• Coordinates of the starting point: Sequence
• Optimization accuracy: Real

Outputs

• Coordinates of the extremum point
  Point(TypedDict):
  point: Sequence[Real]
  f_value: Real
• History (TypedDict):
  iteration: List[Integral]
  f_value: List[Real]
  x: List[Sequence]
  message: AnyStr

Methods

• Newton’s method under equality constrains

• Log Barrier Method
• Primal-Dual Interior-Point Methods

Reference

https://www.cs.cmu.edu/~pradeepr/convexopt/Lecture_Slides/primal-dual.pdf (Primal-Dual Interior-Point Methods)
https://www.stat.cmu.edu/~ryantibs/convexopt-S15/scribes/15-barr-method-scribed.pdf ( Log Barrier Method)

https://www.cs.cmu.edu/~ggordon/10725-F12/scribes/10725_Lecture12.pdf (Newton’s method under equality constrains)

Streamlit app

Input requirements

• Method
• Function
• Equalities limitations
• Inequalities limitations
• Start point
• Accuracy

Output requirements

• Problem
• Solutions(X_min, f(x)_min, iter)
• 3d plot
• Contour lines plot

Brent's method vis

Create documentation

Gradient descent

Developing gradient descent

• constant step
• fractional step
• optimal step

Brent's method develop\

BFGS algorithm vis

Streamlit App

Input requirements

• Get an excel or csv file that has X and y in it
• Get a name of column y
• Type of regression
• Degree of polynomial regression
• Optimization algorithm (Calculate using matrices or gradient methods)
• button with regulators type
• choose x and y columns

Output requirements

• Basic regression metrics
• Excel file available for download

BUGS

Lambdify takes 2 args but 4 was given

2022-05-0

5 10:26:21.678 Uncaught app exception

Traceback (most recent call last):

File "/home/appuser/venv/lib/python3.9/site-packages/streamlit/scriptrunner/script_runner.py", line 443, in _run_script

exec(code, module.__dict__)

File "/app/optimization_ml/Inner_point/streamlit_app.py", line 101, in

solution, history = bound_constrained_lagrangian_method(function.call, x0, constraints, max_iter=100)

File "/app/optimization_ml/Inner_point/algorithms.py", line 87, in bound_constrained_lagrangian_method

constraints[i](x0)

File "/app/optimization_ml/StreamlitSupport/functions.py", line 39, in function_callable

return function_callable_initial(*x)

TypeError: _lambdifygenerated() takes 2 positional arguments but 4 were given

Nonlinear conjugate gradient method

Make Nonlinear conjugate gradient method

Using documentation multidimensionaloptimization.pdf page 6, algorithm 2.4.1 make new method.

nonlinear_cgm.py

• solving core
• verbose printing iteration
• history object
• test on some function
• formatting a docstring

support.py

• consider creating a new object in the history for the method

Create input handler

Create a handler from string data

Using sympy https://docs.sympy.org/latest/modules/parsing.html create a raw data parser from a string.

List of functions that the parser must read

• exponential function
• logarithms
• ...

This list will be updated

Streamlit app

Required input fields

• Type of algorithm. (Gradient with constant step, fractional step, optimal step, nonlinear conjugate gradient method )
• Field with function input 𝑓(x)
• Field with the analytical gradient flag. If analytical gradient flag is True then request the gradient in analytical form. ∇𝑓(x)
• Field with the start point (The default value is random numbers from a uniform distribution on (-1, 1))
• Search precision 𝜀
• Field with a history saving flag
• Required parameters for each method

Output:

• Found point
• plots

Refactor path way

Change path

1. Change the path so that the streamlit application works from the optimization_ml/Regression directory
   

   optimization_ml/Regression/streamlit_app.py

   Line 5 in fbb4fcc

   sys.path.insert(0, os.path.abspath('.'))

2. Move the pink theme .streamlit to the optimization_ml/Regression directory
3. Change color for MDOpt
   The main purpose is to create streaming applications with different colors from the same repository

BFGS algorithm dev

Streamlit. 1d algorithms

Create the streamlit page with algorithms.

Left big task (on grey background)

First item

• Funciton parser #2

Second item #10

• min-max button
• bounds request
• type optimization
• verbose
• rows requset
• calculate button

Right big task (main part)

• Show enterned info
• create plot
• print verbose (or no)
• button with create report about algs

Create a mathematical model of MultiDim oprimization