eve-tools / element43 Goto Github PK
View Code? Open in Web Editor NEWMarket, Trade and Industry Manager for EVE Online
License: BSD 3-Clause "New" or "Revised" License
Market, Trade and Industry Manager for EVE Online
License: BSD 3-Clause "New" or "Revised" License
Absolute minimum tables needed from static data dump in the short term:
invTypes
mapRegions
mapSolarSystems
invMetaGroups
invMarketGroups
staStations
That should be enough to get us rolling.
It takes the page about 30 seconds to load. Performance has to be increased drastically.
I'm trying to run e43 on my VPS which has 2 GB memory (~1,7 free). While processing the 'ramAssemblyLineTypeDetailPerGroup' table the script stops with the following error.
django.db.transaction.TransactionManagementError: Transaction managed block ended with pending COMMIT/ROLLBACK
I also had some other errors which I can't remember yet, but the fact, that this script is using my whole memory should be fixed.
Whenever esi-markets detects an update of an order within a market in a region it saves the matching combination of regionID and typeID. At the moment this mechanism is only used for logging. This describes an endpoint for streaming these combinations as updates occur. Clients like vue43 could then update their local view when receiving such a region/type pair resulting in 'real-time' display of market data. Also, more advanced market statistics could be reactively generated in downstream services such as market-stats (95th percentile...). We could even use that change detection mechanism for implementing an order watching and notification service.
There already is dummy code in esi-markets for this. The 'source' stream could be added to the store or its external interface could be used. Given the fact that the gRPC implementation already has access to the store for querying orders it would be easy to add the source stream to the store instance, too.
At the moment there is no retry mechanism for the first request (for getting X-Pages) when fetching orders from ESI for a region/structure. If that request fails, the whole region won't get an update. A retry mechanism should be added here and here. Code can be copied from here. However if we want to do this The Right Way™ we should perform a randomized exponential backoff in those cases.
Right now we can only view the profit we made with a certain type over the complete timespan of our data. An ability to view the profits of e.g. the last 7 days should be useful.
We should add Prometheus metrics collection to jumpgate so we can analyze the system's usage and performance.
Get to work, fools!
At the moment there is no real documentation on how to get started with development. Also a simple to set up dev environment is missing. We need documentation detailing a convenient dev workflow on:
While modifications to the frontend can be done locally by simply using the production API (and disabling CORS in the browser), developing new services or modifying existing ones requires a local instance of the backend (if the service needs to access other services in the cluster). We're using Kubernetes in production so it would only be fitting to use that for our dev environment, too. Individual YAML manifests for each service could be deployed to a local minikube instance or an arbitrary Kubernetes cluster for development while telepresence is used for proxying traffic of the process on the devs machine into the cluster.
While running our own CI system can be fun, it is yet another service to maintain. We could migrate to one of the numerous CI platforms which offer free services for open-source projects.
It looks like something's an issue with the responsive design — what you get is exactly what you get on the desktop when window width is < 768 px. Rotating to landscape has no effect even though my display should have enough pixels to trigger the next breakpoint.
It would be pretty rad if you could go from an item detail page like this: http://www.element-43.com/market/17363/
To a trade finder to pre-configured for the item in question. This could be a pretty neat thing if done correctly. The current tradefinder requires you to enter a start/end region, which is appropriate for the more general case.
For a specific item, perhaps we do a bunch of cross-region comparing and find the best route. They'd still need to filter out some obviously shite routes, and we may need to give them some additional settings to tweak what gets returned (perhaps we're not interested in low/nullsec routes).
My complaints related to the search bar:
Add the ability to filter Journal Entries by refType.
If someone is interested in a deep-dive into memory allocation internals, this one is for you. Statically built versions of esi-markets tend to use way more memory that their dynamically linked counterparts. The repository contains instructions on how to build a static version of esi-markets which can be used for profiling etc.. How can we get a statically-linked version which uses less memory at runtime?
Vue43 depends on many NPM packages. It would be great if we could update them automagically by using something like greenkeeper.
The BoM for the Procurer lists all required minerals two times with different amounts. Doesn't happen with all blueprints.
This happens for example with the Augoror and the reason is the changed BoM since Retribution. The old algorithm to determine if it's "advanced" or "non-advanced" manufacturing doesn't work anymore.
Now that we have switched to gRPC for our cluster's internal communication we can generate OpenAPI docs by using grpc-gateway on which jumpgate is based. The individual service's Protobuf annotations are there, we just need to combine the individual generated Swagger docs using something like swagger-combine and package them up for deployment with swagger-ui. We may even be able to use parts of esi-swagger-ui so our docs would look like ESI's documentation.
We could generate and display some stats from ESI market's data store such as:
Any more ideas?
In cases where the EVE API is unavailable and thus slowing down the requests or when there are not enough worker processes available to handle all the scheduled tasks, the task queue will keep growing, because Celery will automatically add duplicate tasks.
This behavior should be prevented.
Task apps.api.tasks.ProcessResearch[*] raised exception: AttributeError('reserachStartDate',)
api/tasks.py:
start_date=pytz.utc.localize(datetime.datetime.utcfromtimestamp(job.reserachStartDate)),
probably a typo - will be fixed shortly.
I'd suggest prioritizing PLEX in the ajax search when entering 'plex'. Right now you get stuff like Complex Organisms and Complex Reactor Blueprint, but not Pilot License Extension (which is what most people are probably expecting when they enter 'plex').
Initial install seemed to go fine but no joy at the very end, no price data to DB as near as I can tell. Maybe this will help explain:
I wanted clean logs and such so I restarted the VM and now I can't figure out how to connect with supervisorctl at all. Do I need to source into a virtualenv or something? Lost.
gtaylor said you've got finals, good luck with those and THANKS
Hi,
When i try to build the docker image i get this error:
ERROR in /vue43/components/Search.vue.ts
[tsl] ERROR in /vue43/components/Search.vue.ts(63,7)
TS2322: Type 'string | (string | null)[]' is not assignable to type 'string'.
Type '(string | null)[]' is not assignable to type 'string'.
ERROR in ./node_modules/css-loader??ref--6-1!./node_modules/postcss-loader/lib??ref--6-2!./node_modules/stylus-loader??ref--6-3!./stylus/main.styl
Module build failed: Error: /vue43/node_modules/vuetify/src/stylus/components/_calendar-daily.styl:5:40
1| @import '../bootstrap'
2| @import '../theme'
3|
4| v-calendar-daily($material)
5| background-color: $material.calendar.background-color
---------------------------------------------^
6| .v-calendar-daily__intervals-head
7| border-right: $material.calendar.line-color 1px solid;
8| .v-calendar-daily_head-day
$material.calendar has no property .background-color
at v-calendar-daily() (/vue43/node_modules/vuetify/src/stylus/components/_calendar-daily.styl:5:1)
at ".theme--dark.v-calendar-daily" (/vue43/node_modules/vuetify/src/stylus/theme.styl:13:3)
at dark() (/vue43/node_modules/vuetify/src/stylus/theme.styl:12:1)
at theme() (/vue43/node_modules/vuetify/src/stylus/theme.styl:4:1)
at CachedPathEvaluator.Evaluator.visitMember (/vue43/node_modules/stylus/lib/visitor/evaluator.js:351:11)
at CachedPathEvaluator.Visitor.visit (/vue43/node_modules/stylus/lib/visitor/index.js:28:40)
at CachedPathEvaluator.Evaluator.visit (/vue43/node_modules/stylus/lib/visitor/evaluator.js:160:18)
at CachedPathEvaluator.Evaluator.visitExpression (/vue43/node_modules/stylus/lib/visitor/evaluator.js:644:26)
at CachedPathEvaluator.Visitor.visit (/vue43/node_modules/stylus/lib/visitor/index.js:28:40)
at CachedPathEvaluator.Evaluator.visit (/vue43/node_modules/stylus/lib/visitor/evaluator.js:160:18)
at CachedPathEvaluator.Evaluator.visitExpression (/vue43/node_modules/stylus/lib/visitor/evaluator.js:644:26)
at CachedPathEvaluator.Visitor.visit (/vue43/node_modules/stylus/lib/visitor/index.js:28:40)
at CachedPathEvaluator.Evaluator.visit (/vue43/node_modules/stylus/lib/visitor/evaluator.js:160:18)
at CachedPathEvaluator.Evaluator.visitProperty (/vue43/node_modules/stylus/lib/visitor/evaluator.js:687:22)
at CachedPathEvaluator.Visitor.visit (/vue43/node_modules/stylus/lib/visitor/index.js:28:40)
at CachedPathEvaluator.Evaluator.visit (/vue43/node_modules/stylus/lib/visitor/evaluator.js:160:18)
at CachedPathEvaluator.Evaluator.visitBlock (/vue43/node_modules/stylus/lib/visitor/evaluator.js:720:39)
at CachedPathEvaluator.Visitor.visit (/vue43/node_modules/stylus/lib/visitor/index.js:28:40)
at CachedPathEvaluator.Evaluator.visit (/vue43/node_modules/stylus/lib/visitor/evaluator.js:160:18)
at CachedPathEvaluator.Evaluator.invoke (/vue43/node_modules/stylus/lib/visitor/evaluator.js:1087:17)
@ ./stylus/main.styl 4:14-187
@ ./store/index.ts
@ ./store ^\.\/(?!-)[^.]+\.(js|ts)$
@ ./.nuxt/store.js
@ ./.nuxt/index.js
@ ./.nuxt/client.js
ERROR
Error: Webpack build exited with errors
I try to run the services using docker-compose. Here is my docker-compose.yml:
version : "3.7"
services:
vue:
build: https://github.com/EVE-Tools/vue43.git
container_name: vue43
restart: unless-stopped
esi-markets:
build: https://github.com/EVE-Tools/esi-markets.git
container_name: esi-markets
environment:
- REFRESH_TOKEN=
- CLIENT_ID=
- SECRET_KEY=
- GRPC_HOST=43000
networks:
- gateway
restart: unless-stopped
static-data:
build: https://github.com/EVE-Tools/static-data.git
container_name: static-data
environment:
- LOG_LEVEL=info
- PORT=43000
- DB_PATH=static-data.db
- ESI_HOST=esi.tech.ccp.is
- STRUCTURE_HUNT_HOST=stop.hammerti.me.uk
- DISABLE_TLS=false
networks:
- gateway
restart: unless-stopped
top-stations:
build: https://github.com/EVE-Tools/top-stations.git
container_name: top-stations
environment:
- CRON=@hourly
- DB_PATH=/data/top-stations.db
- LOG_LEVEL=info
- ESI_MARKETS_HOST=esi-markets:43000
- PORT=43000
networks:
- gateway
restart: unless-stopped
market-stats:
build: https://github.com/EVE-Tools/market-stats.git
container_name: market-stats
environment:
- SEED_DB=false
- CRON=0 52 * * *
- LOG_LEVEL=info
- POSTGRES_URL=postgres://market-stats@localhost:5432/market-stats?sslmode=disable
- PORT=43000
networks:
- gateway
restart: unless-stopped
jumpgate:
build: https://github.com/EVE-Tools/jumpgate.git
container_name: jumpgate
environment:
- LOG_LEVEL=
- LISTEN_HOST=:8000
- ESI_MARKETS_HOST=esi-markets:43000
- MARKET_STATS_HOST=market-stats:43000
- STATIC_DATA_HOST=static-data:43000
- TOP_STATIONS_HOST=top-stations:43000
ports:
- 8000:8000
restart: unless-stopped
networks:
gateway:
When a user updates a price for a material the price should be saved in the session or maybe in the browser (html5).
This way a user doesn't need to re-enter prices if he selects another blueprint or just reloads the page.
The price should be saved as soon as the user leaves the field.
Go here http://www.element-43.com/market/29668 and click between Orders and Regions tab. The page length extends in Orders to the point of needing a vertical scroller, whereas Regions doesn't.
Probably best to always show the scrollbars to avoid the horizontal bouncing.
Automatically use HTTPS for login/auth and wallet stuff. Also serve assets via HTTPS.
The table 'api_skill' and 'api_skillgroup' are empty and this generate 'violates foreign key constraint' if I insert an new api key.
Would be great to see the most benefitial item for every LP Store.
Show the perfect ME an PE next to the corresponding input fields in the manufacturing job form.
Just testing hooks. Ignore this.
It seems like expenses are not taken into account.
At the moment the Bill of Materials (BoM) only show the raw matierals (also called commodity or resource) and/or highest tier components (items that can be manufactured) required to manufacture the item. When the BoM contains components it is not possible to see their BoM.
The manufacturing tool should therefor allow the user to display the BoM of all components that are required by the blueprint.
Some sort of tree view should be the best way to implement this. The user should be able to decide which components BoM he wants added to the overall BoM.
At the moment it is a ~1kloc blob of Go code in a single file which needs to be split up into individual packages in preparation for reactive stat generation made possible by #46.
User registers, gets the link in the email and clicks on it, gets a 404 page from the website and user is not activated.
I just manually activated a user because of it. Also I believe barbers had the same problem.
The readme is pretty minimalistic compared to what is going on in the code. A section describing the structure of the service should be added.
The entire page is being cached (including the username at the top right) - we should implement partial caching.
Right now the performance in the "Browse Market" view is not great. The market groups are loaded dynamically on every request.
To improve performance all market groups could be "pre-compiled" in a simple JSON file. The format of the JSON file will be provided by @zweizeichen. To create the JSON I propose a Django command that can be run from the command line.
The only drawback to that solution is that the file has to be "compiled" everytime the database dump is imported.
@gtaylor proposed to generate the file in the view and just cache it or have it generated periodically with a celery task. Either way, this would avoid the drawback described before.
http://i.imgur.com/bgHBTNH.png
Looks like something Bootstrap-related may be off. Notice the square placeholder character at the left of the alert div.
After you enter your API ID and Verification ID you can select the characters you want to import. Hitting the "Add Character" button results in an Django exception:
Environment:
Request Method: POST
Request URL: http://element-43.com/settings/api/character/*/*/
Django Version: 1.4
Python Version: 2.7.3
Installed Applications:
('django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.messages',
'django.contrib.staticfiles',
'django.contrib.admin',
'django.contrib.admindocs',
'django.contrib.humanize',
'south',
'devserver',
'apps.market_data',
'apps.api',
'eve_db')
Installed Middleware:
('django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware')
Traceback:
File "/opt/python27/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
111. response = callback(request, *callback_args, **callback_kwargs)
File "/opt/python27/lib/python2.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
20. return view_func(request, *args, **kwargs)
File "/home/app/element43/webapp/element43/apps/market_data/views/settings.py" in api_character
159. if len(key_info.key.expires) == 0:
Exception Type: TypeError at /settings/api/character/*/*/
Exception Value: object of type 'int' has no len()
Same as in the ask/bid view - we are experiencing extreme performance issues here, too.
The order overview should allow you to compare your ask/bid orders which are on the same station to support station traders.
We're using CREST for generating our sitemap. This is bad and we should/have to move to ESI before CREST gets taken offline this spring. The way ESI is designed we need to do a little more work for determining market types but it is doable.
The flate2 crate used for compressing esi-market's persistence is outdated. It should be upgraded to the latest version and some simple tests should be performed to pick the fastest variant (miniz_oxide vs. miniz etc.).
Related code:
https://github.com/EVE-Tools/esi-markets/blob/6b135fb08b5a421d7885663e285d60214756abee/Cargo.toml#L13-L14
It seems like the quicklook view tends to be confusing - it should be easier to get an instant impression of a given type's prices with a redesigned view.
Need to create the models to handle proper handling of corporate data in the system. Specifically should use character rights from API but be manageable by the CEO/director. This will require a new front-end.
In order to speed up initial loading time for a given type's regional view, its loading should be deferred via AJAX like the region tab in the type view
I often need to make multiple searches. When I search for an Item, it takes me to a new page, with no way to search for another item without clicking back or opening a new Element-43 tab. It would be nice to have a search bar at the top of each items page that opens in a new tab so we could do multiple searches quickly.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.