Scraper#3
Conversation
| SPACES_ACCESS_KEY= | ||
| SPACES_SECRET_KEY= | ||
| SPACES_REGION= | ||
| SPACES_PUBLIC_URL= No newline at end of file |
There was a problem hiding this comment.
are these needed now that the migration of images is done? might be an easy next attack avenue if our server is attacked/hacked
| "scrape": "tsx prisma/scraper.ts", | ||
| "test:scraper": "TEST_MODE=true tsx prisma/scraper.ts", | ||
| "migrate:static-images": "tsx prisma/migrateStaticImages.ts", | ||
| "migrate:static-images:dry-run": "DRY_RUN=true tsx prisma/migrateStaticImages.ts", |
There was a problem hiding this comment.
same with these migrate commands as the above comment
| "format": "prettier --write '{src,test}/**/*.ts'", | ||
| "seed": "tsx prisma/seed.ts" | ||
| "scrape": "tsx prisma/scraper.ts", | ||
| "test:scraper": "TEST_MODE=true tsx prisma/scraper.ts", |
There was a problem hiding this comment.
Are we ever going to run the test to verify anything before running the actual scraper?
| }, | ||
| "homepage": "https://github.com/cuappdev/eatery-blue-backend#readme", | ||
| "dependencies": { | ||
| "@aws-sdk/client-s3": "^3.930.0", |
There was a problem hiding this comment.
If we no longer need the migrating images script, then we should uninstall this dependency
| case 'General': | ||
| return EventType.GENERAL; | ||
| case 'Free Food': | ||
| return EventType.GENERAL; |
There was a problem hiding this comment.
I don't think the "General" or "Free Food" event types ever show up in the dining API. Instead of having them in the mapping, there should be logic to manually add these types to events for that need them (confirm with the rest of the team when these should be used.
There was a problem hiding this comment.
They show up in the static eateries.
| }>; | ||
| }>; | ||
| }>; | ||
| payMethods: Array<{ |
There was a problem hiding this comment.
This is the same as RawPayMethod[] so use that
| nameshort: string; | ||
| about: string; | ||
| aboutshort?: string; | ||
| cornellDining?: boolean; |
There was a problem hiding this comment.
Why are some of the fields optional with the question mark
We control all the information about all static eateries, so nothing would be optional (we should have consistent data for all of the static eateries)
| }>; | ||
| announcements?: string[]; | ||
| diningItems?: Array<{ | ||
| item: string; |
There was a problem hiding this comment.
Can reuse types here - the closer it is to the dining api schema, the less specialized code you have to write and the more you can reuse
| @@ -0,0 +1,51 @@ | |||
| import cron from 'node-cron'; | |||
There was a problem hiding this comment.
the scraper will be run in a separate container than the main app. Therefore, the scheduler part should be in the scraper itself, not in src/...
As of now, there is no way of running the scraper on a schedule (only once using npm run scrape)
| "sourceMap": true | ||
| }, | ||
| "include": ["src/**/*"], | ||
| "include": ["src/**/*", "prisma/**/*"], |
There was a problem hiding this comment.
nice find, it was really annoying to deal with all the error messages when working in the prisma directory
There was a problem hiding this comment.
Why did we delete this? I used it/added to it while testing notis
| name String | ||
| shortName String | ||
| about String | ||
| shortAbout String |
There was a problem hiding this comment.
Should we have some more default values here as a backup or make more nullable? In case of scraper failures
3 participants
Uh oh!
There was an error while loading. Please reload this page.