mixcloud / django-experiments Goto Github PK
View Code? Open in Web Editor NEWDjango AB testing module
License: MIT License
Django AB testing module
License: MIT License
Goals have not been tested in the last few commits.
May already be implemented. Need to check.
If cache is not invalidated, then experiments will not update for users until cache is reset.
Dates style is different when the nexus index page is loaded and when a new experiment is added. This is due to differences in the way the data is rendered. On page load, the QuerySet is rendered into the template directly via django. When a new experiment is added, the data is serialized server side, and de-serialized client side. The de-serialization causes the date to take on a slightly different format to when the data is rendered directly.
It is very messy.
Cannot remember what this was, but it is on the TODO list.
"url/slug base partitioning, for object/page testing. Googlebot based testing"
The nexus dashboard currently only shows experiments that are toggled to global. Need to add support for selectively activated experiments too.
If there are duplicate conditions and the user tries to delete one of them, all will be deleted. The delete function is also ignorant to the excluded variable.
Changes to remove_conditions()
in nexus_modules.py
:
exclude
as a criteria, also make sure to POST
it in js.Cannot set conditions before switching to selective mode as conditions are hidden unless in selective mode.
When a new experiment is added, the page needs to be refreshed before a condition can be added
Need to add a license (see MIT) to the root of the directory.
Was caused in fa82496c2608187b6a55206094ea72458be90e72 when fixing #27. Caused by the use of datetime data in the start_date field.
Note, still works after a page refresh. Check JavaScript and that functions are returning correctly.
Tidy up redis counters when the experiment is deleted. Will currently be stored indefinitely.
Attach a signal and use KEYS/DEL commands: KEYS experiments:<experiment_name>:*
Created in fa82496c2608187b6a55206094ea72458be90e72, which solved #27. "Just Now" is only used to prevent the server from mysteriously hanging when adding a new experiment. It is something to do with returning an experiment.to_dict() which contains a start_date of type DateTime.
If an invalid IP address is used in an IP address condition, the server will not send a response. The error is being caught, however, as it appears in the console log.
start date and end date sorting do not currently do anything.
The project name needs to be changed. Causes some clarity issues with the code, as experiments
is used to loosely.
Post ideas in the comments ;)
Add enabled/disabled states to specific alternatives within an experiment.
Every experiment will record a every goal hit, even if the hit is intended to come from another experiment. This can cause very inaccurate data.
Closely related to #8.
Deleting experiments and enrollments causes major performance issues many db records are deleted at once. In the short term:
In the long term, the way enrollments are handled needs to be looked into.
Give results CSS some much needed beautification.
Add status and edit buttons.
Should be able to change the status/edit the experiment once actually within the results page.
Should come into effect with #12.
Cycle counter does not reset when being used within a for loop, making it difficult to create alternating column classes when dynamically generating them in the results.html table.
Currently using -{% if forloop.counter|divisibleby:2 %}even{% else %}odd{% endif %}
. Waiting for a patch to fix this.
Add the ability to modify/reset goal values through nexus.
Possibly even relocate goals from settings.py to redis.
Improve the nexus front page styling. More information would also be useful here.
If no conditions are met, the user does not get a session for the experiment. If then a condition is added which they match, they will receive session data for the experiment.
Their session data may be different before and after they meet the condition, and so they could jump from the control to another alternative. Is this correct?
i.e. N/A
Was on TODO, can't really remember what this is about =P
Line 175 (as of time of writing). Need to clarify how this code is used and whether it is now depreciated.
Caching or experiment model at runtime to reduce db hits (see gargoyle and modeldict)
Investigate the way conditions are inclusive/exclusive of one another.
Currently, conditions are inclusive of one another within a condition_set
but exclusive between sets.
Add a link/button to quickly show all available goals on the results page.
Many implications. Needs to discuss.
Based on the chi squared test, a goal should only be incremented once per enrollment.
Log the following to file
To do with JS and template.
Caused in 9acb738438a174a2d6f2ed686d6da7e905024f16
When defining alternatives in a template, each alternative hits the server separately. Could be an issue with heavy traffic.
Have forgotten to git add .
in the last few commits so there is a whole load of missing code. Fixing now.
Cannot currently edit the experiment settings (bar conditions) from within the experiment results view. Can either overlay edit experiment form or follow design similar to that suggested in #39.
Would it make sense for a goal to be only incremented once per enrollment?
Enrollments contain experiment foreign keys, and so when experiments are called through the enrollment, i.e.
utils.py line 153: if enrollment.experiment.state == DISABLED_STATE:
the experiment cache is not used. Should replace with the experiment_name
, and use the experimentmanager
cache to retrieve the experiment.
Cannot remember what this is, but it was on the TODO list.
experimentmanager should autocreate the experiment if it doesn't exist. This will prevent keyerrors.
start_time
and end_time
need to be populated automatically. What about scheduling the experiment to end at a certain time?
end_date
is never assigned a value. Should be done when an experiment is disabled.
Allow specific goal tracking in results view.
Should we assume that the experiment state is global
if there are no conditions?
Conditions do not seem to be functioning correctly when only a single "exclude" condition is set.
Tested on not anonymous
and not username
.
The results page should support the collapsing and expanding of alternatives.
This could cause issues with the odd/even classes of columns. Could prevent this issue by shrinking the alternative from 3 columns to one column.
Add JS to toggle goals from within the results table. Similar to condition toggling?
in nexus_modules.py line 165.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.