tylercollier / openresync Goto Github PK

So, I was able to run the project and everything seems to be working great except that it will not fetch any properties.

I have changed the config to this:

const rebnyTrestle = {
  getResourceEndpoint: mlsResourceObj => {
    const endpoint = `https://api-prod.corelogic.com/trestle/odata/${mlsResourceObj.name}`
    const url = new URL(endpoint)
    // These can be useful subsets for dev purposes
    // if (mlsResourceObj.name === 'Property') {
    //   url.searchParams.set('$filter', "City eq 'Georgetown'")
    // } else if (mlsResourceObj.name === 'Member') {
    //   url.searchParams.set('$filter', "MemberCity eq 'Georgetown'")
    // }
    return url.toString()
  }
}

I am getting an error 400 on the sync step

You can find the logs here:

https://justpaste.it/9r0i6

I tried redacting any sensitive information, I have also removed the data field that contained an encoded numbers array.

This is a screenshot from the user interface.

Let me know if you need more information on this issue.

Regards

Cron schedules will be set up by the developer in the config.js file. But we should show them on the dashboard. I believe the npm cron package can show the upcoming exec times.

Like mentioned at https://stackoverflow.com/a/20959426/135101

Breaking out #3 into more cases.

• When did the last purge occur and was it successful?
  • How many records were purged, per MLS resource?

Ideally for sync and purge

We need to keep track of the data that we want to show on the website. I'm calling them stats. I want the user of the website to be able to see answers to questions like these:

• When did the last sync occur and was it successful?
  • How many listings were updated
• Is my data up to date?
  • How many records do I have and how many are in the MLS?
  • What is the most recent listing (this kind of proves if the last sync is when we say it is)
• Get me the data for X record
  • Ideally, the data from my destinations as well as the original in the MLS
  • Perhaps have a way to compare them, or show a diff?
• During a sync, how many have been synced so far, how many to go, and estimate of completion time
  • It'd be fun to know the details, like download time, time per each destination

Happy new year.

Great project you have there. I tried running it but I am getting the following error:

Error: User config version (undefined) does not match software version (0.2.0)

I used the default example config without any modifications just to get it up and running.
Didn't notice any errors during the build process.

The full error that I get when I try to run it wit the following command:

NODE_ENV=production TZ=UTC node server/index.js is:

/root/openresync/lib/config.js:24
      throw new Error(`User config version (${builtUserConfig.version}) does not match software version (${userConfigVersion})`)
      ^

Error: User config version (undefined) does not match software version (0.2.0)
    at buildUserConfig (/root/openresync/lib/config.js:24:13)
    at Object.<anonymous> (/root/openresync/server/index.js:31:20)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12)
    at internal/main/run_main_module.js:17:47

I am not primarily a node developer but let me know if I can help somehow.
I am very interested in this project's development.

Regards

Nested $select in $expand too

Since the ABOR sample dataset is never updated, here's what we'll do to test it. Download the first X pages and stop. This will kind of simulate that more are added, because the next time we run it, we'll have some left to do (until we've got them all).

However, there's a problem. If we decide to stop at X pages, we can't tell if there would have been more records with the same timestamp, and the next time we run and use "gt" to fetch records, we might miss some. So, we'll keep going as long as the ModificationTimestamp doesn't change.

Make the website look acceptable. Get quick design help from a designer.

This repo could fairly easily be published on Docker. I published an image here, but it would be far better if it was handled through automated builds from Github and Docker integration with this repo.

The Dockerfile I used to build the image I published is

# STAGE: INSTALL DEPENDENCIES
FROM node:16-alpine AS deps
# Check https://github.com/nodejs/docker-node/tree/b4117f9333da4138b03a546ec926ef50a31506c3#nodealpine to understand why libc6-compat might be needed.
#RUN apk add --no-cache libc6-compat
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm install

# STAGE: BUILD ARTIFACTS
FROM node:16-alpine AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
COPY config/config.example.js ./config/config.js
RUN npm run build

# STAGE: RUN
FROM node:16-alpine AS runner
WORKDIR /app

ENV NODE_ENV production

#RUN addgroup --system --gid 1001 nodejs
#RUN adduser --system --uid 1001 nodejs

COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/dist ./dist
COPY . .

#USER nodejs

VOLUME ['/app/logs', '/app/config']
EXPOSE 3000

ENV PORT 3000

CMD ["node", "server/index.js"]

A few improvements could be made, but this is good enough for my current use case. In particular, I didn't want to juggle permissions for the volumes between the host that runs the container and the container itself, so the container runs as root.

This image requires a config file to be mounted at /app/config.js. My typical usage in docker-compose.yml looks like this:

  openresync:
    image: ckeeney/openresync
    restart: always
    ports:
      - 4000:4000
    environment:
      TRESTLE_CLIENT_ID: SECRET
      TRESTLE_CLIENT_SECRET: SECRET
      DB_CONN_STRING_TRESTLE: mysql://root:root@mysql:3306/re-data
      DB_CONN_STRING_STATS: mysql://root:root@mysql:3306/re-data
    volumes:
      - ./volumes/openresync/config:/app/config
      - ./volumes/openresync/logs:/app/logs
    depends_on:
      - mysql

Down the road, it would also be nice to bundle a config file that looks at a bunch of different environment variables so it could be possible to run the image by setting environment variables without the need to write an entire config file. Supporting things like selecting fields makes this fairly complicated so it's probably fine to not focus on this feature until later.