GithubHelp home page GithubHelp logo

kan01234 / ur-web-spider Goto Github PK

View Code? Open in Web Editor NEW
5.0 5.0 1.0 36.53 MB

web spider to scan UR avialbe room and output as csv

License: MIT License

Python 100.00%
csv html-parsing json python web-crawler web-spider

ur-web-spider's Introduction

About Me:

  • 👋 Hi, I’m @kan01234
  • 😀 Can speak English, Mandarin, Cantonese, Japanese
  • 👨‍💻 Wokring as Software Engineer, Full stack Developer
  • 🏠 Living in Japan🇯🇵 now
  • 🚪 Please check my portfolio
  • 📫 How to reach me

safghjkl@gmail.com LinkedIn


🔨 Skills

Java  Kotlin  python  TypeScript  JavaScript  Spring  nuxtjs  vuejs  Kafka  CSS  HTML  Firebase  MySQL  MongoDB  NodeJS  AWS  Gradle  npm  Docker  kubernetes  jenkins  nginx 

Top Langs


My Stats

My GitHub stats

GitHub Streak

ur-web-spider's People

Contributors

kan01234 avatar pqab avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar

Forkers

pqab

ur-web-spider's Issues

[bug] parse system

actual

[{'制度_IMG': 'btn_u35.png', '制度名': 'U35割', '制度HTML': 'u35'}]

expect

["U35割"]

[feature] add total time by bus in csv

  • move existing information to new field traffic
  • add field min. total minutes by bus
    • return first one if having same value
  • add nearest station by bus

input

JR**線「高尾」駅バス7分 徒歩1~11分
JR**線「高尾」駅バス7分 徒歩1~11分
JR**線ほか「八王子」駅 徒歩13分

output

18分, JR**線「高尾」駅

[feature] logger

Is your feature request related to a problem? Please describe.
add logger to print the log instead of stdout

Describe the solution you'd like

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[bug] codeQL BADGE should remove

Describe the bug
A clear and concise description of what the bug is
Codeql workflow is disabled and the badge status shown failed in README

To Reproduce
response from the website

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

[bug] the rooms after page 1 is not read

Describe the bug
room after page 1 is not included in csv
螢幕截圖 2022-12-12 11 23 28

To Reproduce
just re-run the program

Expected behavior
room after page 1 should be read to csv

Additional context
update request body pageIndex

curl 'https://chintai.sumai.ur-net.go.jp/chintai/api/bukken/detail/detail_bukken_room/' -X POST -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:107.0) Gecko/20100101 Firefox/107.0' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Accept-Language: zh-TW,zh;q=0.8,en-US;q=0.5,en;q=0.3' -H 'Accept-Encoding: gzip, deflate, br' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Origin: https://www.ur-net.go.jp' -H 'Connection: keep-alive' -H 'Referer: https://www.ur-net.go.jp/' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-site' --data-raw 'rent_low=&rent_high=&floorspace_low=&floorspace_high=&shisya=30&danchi=239&shikibetu=0&newBukkenRoom=&orderByField=0&orderBySort=0&pageIndex=2&pageIndex=2&sp='

[feature] add min. time by walk in csv

  • move existing information to new field traffic
  • add field min. total minutes by walk
    • return first one if having same value
  • add nearest station by walk

input

京王相模原線「京王堀之内」駅 徒歩4~8分
京王相模原線「南大沢」駅バス15分 徒歩4~8分
小田急多摩線「唐木田」駅 徒歩25~28分

output

8分, 京王相模原線「京王堀之内」駅

[feature] update README

Is your feature request related to a problem? Please describe.
update README or related documentation

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[feature] convert bukken json to csv

csv output

  • building name
  • building address
  • building nearest station
  • available room and related information of room

example json

{
    "madori": null,
    "allCount": "15",
    "bukkenCount": "21",
    "roomCount": "8",
    "bukkenImg": "https://chintai.sumai.ur-net.go.jp/chintai/img_photo/20/20_260/20_260_photo_s.jpg",
    "pageIndex": "0",
    "rowMax": "10",
    "pageMax": "5",
    "rentMin": null,
    "rentMax": null,
    "station": [],
    "block": "kanto",
    "tdfk": "tokyo",
    "shisya": "20",
    "danchi": "260",
    "shikibetu": "0",
    "danchiNm": "\u9928\u30f6\u4e18",
    "traffic": "JR\u4e2d\u592e\u7dda\u300c\u9ad8\u5c3e\u300d\u99c5\u30d0\u30b97\u5206 \u5f92\u6b691\uff5e11\u5206<br>\u4eac\u738b\u9ad8\u5c3e\u7dda\u300c\u9ad8\u5c3e\u300d\u99c5\u30d0\u30b97\u5206 \u5f92\u6b691\uff5e11\u5206<br>JR\u4e2d\u592e\u672c\u7dda\u300c\u9ad8\u5c3e\u300d\u99c5 \u5f92\u6b6929\uff5e38\u5206",
    "trafficpdf": "20_2600_traffic.pdf",
    "place": "\u516b\u738b\u5b50\u5e02\u9928\u753a1097",
    "shikikin": "2\u304b\u6708",
    "requirement": "\u30ca\u30b7",
    "kouzou": "\u9244\u7b4b\u30b3\u30f3\u30af\u30ea\u30fc\u30c8\u9020",
    "floorAll": "11",
    "shopBlock": "kanto",
    "shopTdfk": "tokyo",
    "shopHtmlName": "hachioji",
    "shopName": "UR\u8cc3\u8cb8\u30b7\u30e7\u30c3\u30d7\u516b\u738b\u5b50",
    "shopNum": "042-646-6844",
    "shopOpentime": "9\uff1a30\uff5e18\uff1a00",
    "shopHoliday": "\u6c34\u66dc\u3001\u5e74\u672b\u5e74\u59cb\uff0812/29\uff5e1/3\uff09",
    "kiboRoom": [
        {
            "roomLinkPc": "/chintai/kanto/tokyo/20_5210_room.html?JKSS=000010207",
            "roomLinkSp": "/chintai/sp/kanto/tokyo/20_5210_room.html?JKSS=000010207",
            "roomImg": "https://chintai.sumai.ur-net.go.jp/chintai/img_photo/20/20_521/20_521_photo_s.jpg",
            "bukkenNm": "\u30b0\u30ea\u30fc\u30f3\u30cf\u30a4\u30c4\u6b66\u8535\u5883\u901a\u308a",
            "building": "1",
            "roomNo": "207",
            "id": null,
            "year": null,
            "name": null,
            "shikikin": null,
            "requirement": null,
            "madori": null,
            "rent": "120,100\u5186",
            "rent_normal": null,
            "rent_normal_css": null,
            "commonfee": null,
            "commonfee_sp": null,
            "status": null,
            "type": "1LDK",
            "floorspace": null,
            "floor": null,
            "urlDetail": null,
            "urlDetail_sp": null,
            "feature": null
        },
        {
            "roomLinkPc": "/chintai/kanto/tokyo/20_2070_room.html?JKSS=040215403",
            "roomLinkSp": "/chintai/sp/kanto/tokyo/20_2070_room.html?JKSS=040215403",
            "roomImg": "https://chintai.sumai.ur-net.go.jp/chintai/img_photo/20/20_207/20_207_photo_s.jpg",
            "bukkenNm": "\u591a\u6469\u30cb\u30e5\u30fc\u30bf\u30a6\u30f3 \u6c38\u5c71\uff08\u6c38\u5c71\u4e8c\u4e01\u76ee\uff09",
            "building": "4-2-15",
            "roomNo": "403",
            "id": null,
            "year": null,
            "name": null,
            "shikikin": null,
            "requirement": null,
            "madori": null,
            "rent": "55,900\u5186",
            "rent_normal": null,
            "rent_normal_css": null,
            "commonfee": null,
            "commonfee_sp": null,
            "status": null,
            "type": "3DK",
            "floorspace": null,
            "floor": null,
            "urlDetail": null,
            "urlDetail_sp": null,
            "feature": null
        },
        {
            "roomLinkPc": "/chintai/kanto/tokyo/20_2700_room.html?JKSS=002010408",
            "roomLinkSp": "/chintai/sp/kanto/tokyo/20_2700_room.html?JKSS=002010408",
            "roomImg": "https://chintai.sumai.ur-net.go.jp/chintai/img_photo/20/20_270/20_270_photo_s.jpg",
            "bukkenNm": "\u8eca\u8fd4",
            "building": "2-1",
            "roomNo": "408",
            "id": null,
            "year": null,
            "name": null,
            "shikikin": null,
            "requirement": null,
            "madori": null,
            "rent": "89,400\u5186",
            "rent_normal": null,
            "rent_normal_css": null,
            "commonfee": null,
            "commonfee_sp": null,
            "status": null,
            "type": "1LDK",
            "floorspace": null,
            "floor": null,
            "urlDetail": null,
            "urlDetail_sp": null,
            "feature": null
        },
        {
            "roomLinkPc": "/chintai/kanto/tokyo/20_4500_room.html?JKSS=000020115",
            "roomLinkSp": "/chintai/sp/kanto/tokyo/20_4500_room.html?JKSS=000020115",
            "roomImg": "https://chintai.sumai.ur-net.go.jp/chintai/img_photo/20/20_450/20_450_photo_s.jpg",
            "bukkenNm": "\u591a\u6469\u30cb\u30e5\u30fc\u30bf\u30a6\u30f3 \u30d5\u30a1\u30a4\u30f3\u30d2\u30eb\u3044\u306a\u304e \u30d3\u30e5\u30fc\u30d7\u30e9\u30b6\u5411\u967d\u53f0",
            "building": "2",
            "roomNo": "115",
            "id": null,
            "year": null,
            "name": null,
            "shikikin": null,
            "requirement": null,
            "madori": null,
            "rent": "86,500\u5186",
            "rent_normal": null,
            "rent_normal_css": null,
            "commonfee": null,
            "commonfee_sp": null,
            "status": null,
            "type": "1LDK",
            "floorspace": null,
            "floor": null,
            "urlDetail": null,
            "urlDetail_sp": null,
            "feature": null
        },
        {
            "roomLinkPc": "/chintai/kanto/tokyo/20_6771_room.html?JKSS=008040404",
            "roomLinkSp": "/chintai/sp/kanto/tokyo/20_6771_room.html?JKSS=008040404",
            "roomImg": "https://chintai.sumai.ur-net.go.jp/chintai/img_photo/20/20_677/20_677_photo_s.jpg",
            "bukkenNm": "\u3072\u3070\u308a\u304c\u4e18\u30d1\u30fc\u30af\u30d2\u30eb\u30ba",
            "building": "8-4",
            "roomNo": "404",
            "id": null,
            "year": null,
            "name": null,
            "shikikin": null,
            "requirement": null,
            "madori": null,
            "rent": "110,100\u5186",
            "rent_normal": null,
            "rent_normal_css": null,
            "commonfee": null,
            "commonfee_sp": null,
            "status": null,
            "type": "2DK",
            "floorspace": null,
            "floor": null,
            "urlDetail": null,
            "urlDetail_sp": null,
            "feature": null
        },
        {
            "roomLinkPc": "/chintai/kanto/tokyo/20_4460_room.html?JKSS=000010303",
            "roomLinkSp": "/chintai/sp/kanto/tokyo/20_4460_room.html?JKSS=000010303",
            "roomImg": "https://chintai.sumai.ur-net.go.jp/chintai/img_photo/20/20_446/20_446_photo_s.jpg",
            "bukkenNm": "\u753a\u7530\u3082\u307f\u3058\u53f0",
            "building": "1",
            "roomNo": "303",
            "id": null,
            "year": null,
            "name": null,
            "shikikin": null,
            "requirement": null,
            "madori": null,
            "rent": "77,600\u5186",
            "rent_normal": null,
            "rent_normal_css": null,
            "commonfee": null,
            "commonfee_sp": null,
            "status": null,
            "type": "3LDK",
            "floorspace": null,
            "floor": null,
            "urlDetail": null,
            "urlDetail_sp": null,
            "feature": null
        },
        {
            "roomLinkPc": "/chintai/kanto/tokyo/20_2060_room.html?JKSS=305010404",
            "roomLinkSp": "/chintai/sp/kanto/tokyo/20_2060_room.html?JKSS=305010404",
            "roomImg": "https://chintai.sumai.ur-net.go.jp/chintai/img_photo/20/20_206/20_206_photo_s.jpg",
            "bukkenNm": "\u591a\u6469\u30cb\u30e5\u30fc\u30bf\u30a6\u30f3 \u8acf\u8a2a\uff08\u30b0\u30ea\u30fc\u30f3\u30e1\u30be\u30f3\u8acf\u8a2a\uff0d\uff12\uff09",
            "building": "3-5-1",
            "roomNo": "404",
            "id": null,
            "year": null,
            "name": null,
            "shikikin": null,
            "requirement": null,
            "madori": null,
            "rent": "76,600\u5186",
            "rent_normal": null,
            "rent_normal_css": null,
            "commonfee": null,
            "commonfee_sp": null,
            "status": null,
            "type": "3DK",
            "floorspace": null,
            "floor": null,
            "urlDetail": null,
            "urlDetail_sp": null,
            "feature": null
        },
        {
            "roomLinkPc": "/chintai/kanto/tokyo/20_1930_room.html?JKSS=001008405",
            "roomLinkSp": "/chintai/sp/kanto/tokyo/20_1930_room.html?JKSS=001008405",
            "roomImg": "https://chintai.sumai.ur-net.go.jp/chintai/img_photo/20/20_193/20_193_photo_s.jpg",
            "bukkenNm": "\u85e4\u306e\u53f0",
            "building": "1-8",
            "roomNo": "405",
            "id": null,
            "year": null,
            "name": null,
            "shikikin": null,
            "requirement": null,
            "madori": null,
            "rent": "54,400\u5186",
            "rent_normal": null,
            "rent_normal_css": null,
            "commonfee": null,
            "commonfee_sp": null,
            "status": null,
            "type": "3K",
            "floorspace": null,
            "floor": null,
            "urlDetail": null,
            "urlDetail_sp": null,
            "feature": null
        }
    ],
    "system": [
        {
            "\u5236\u5ea6_IMG": "btn_u35.png",
            "\u5236\u5ea6\u540d": "U35\u5272",
            "\u5236\u5ea6HTML": "u35"
        },
        {
            "\u5236\u5ea6_IMG": "btn_kinkyo.png",
            "\u5236\u5ea6\u540d": "\u8fd1\u5c45\u5272",
            "\u5236\u5ea6HTML": "kinkyo"
        },
        {
            "\u5236\u5ea6_IMG": "btn_kinkyo_wide.png",
            "\u5236\u5ea6\u540d": "\u8fd1\u5c45\u5272WIDE",
            "\u5236\u5ea6HTML": "kinkyo_wide"
        }
    ],
    "room": [
        {
            "shisya": "20",
            "danchi": "260",
            "shikibetu": "0",
            "roomLinkPc": "/chintai/kanto/tokyo/20_2600_room.html?JKSS=003100503",
            "roomLinkSp": "/chintai/sp/kanto/tokyo/20_2600_room.html?JKSS=003100503",
            "roomNmMain": "3-10\u53f7\u68df",
            "roomNmSub": "503\u53f7\u5ba4",
            "system": [
                {
                    "\u5236\u5ea6_IMG": "btn_u35.png",
                    "\u5236\u5ea6\u540d": "U35\u5272",
                    "\u5236\u5ea6HTML": "u35"
                }
            ],
            "allCount": "8",
            "pageIndexRoom": "0",
            "rowMaxNext": "10",
            "rowMax": "5",
            "allRoomUrl": "/chintai/kanto/tokyo/20_2600.html",
            "design": [],
            "featureParam": [],
            "id": "003100503",
            "year": null,
            "name": null,
            "shikikin": null,
            "requirement": null,
            "madori": "https://chintai.sumai.ur-net.go.jp/chintai/img_madori/20/20_260/20_260_0-03-0010_3Bm_RA_01_00008_s.gif",
            "rent": "43,400\u5186",
            "rent_normal": "",
            "rent_normal_css": " dn",
            "commonfee": "3,300\u5186",
            "commonfee_sp": null,
            "status": null,
            "type": "3K",
            "floorspace": "48&#13217;",
            "floor": "5\u968e",
            "urlDetail": null,
            "urlDetail_sp": null,
            "feature": null
        },
        {
            "shisya": "20",
            "danchi": "260",
            "shikibetu": "0",
            "roomLinkPc": "/chintai/kanto/tokyo/20_2600_room.html?JKSS=003030406",
            "roomLinkSp": "/chintai/sp/kanto/tokyo/20_2600_room.html?JKSS=003030406",
            "roomNmMain": "3-3\u53f7\u68df",
            "roomNmSub": "406\u53f7\u5ba4",
            "system": [
                {
                    "\u5236\u5ea6_IMG": "btn_u35.png",
                    "\u5236\u5ea6\u540d": "U35\u5272",
                    "\u5236\u5ea6HTML": "u35"
                }
            ],
            "allCount": "8",
            "pageIndexRoom": "0",
            "rowMaxNext": "10",
            "rowMax": "5",
            "allRoomUrl": "/chintai/kanto/tokyo/20_2600.html",
            "design": [],
            "featureParam": [],
            "id": "003030406",
            "year": null,
            "name": null,
            "shikikin": null,
            "requirement": null,
            "madori": "https://chintai.sumai.ur-net.go.jp/chintai/img_madori/20/20_260/20_260_0-03-0003_3Bm_RA_01_00009_s.gif",
            "rent": "44,300\u5186",
            "rent_normal": "",
            "rent_normal_css": " dn",
            "commonfee": "3,300\u5186",
            "commonfee_sp": null,
            "status": null,
            "type": "3K",
            "floorspace": "48&#13217;",
            "floor": "4\u968e",
            "urlDetail": null,
            "urlDetail_sp": null,
            "feature": null
        },
        {
            "shisya": "20",
            "danchi": "260",
            "shikibetu": "0",
            "roomLinkPc": "/chintai/kanto/tokyo/20_2600_room.html?JKSS=001120417",
            "roomLinkSp": "/chintai/sp/kanto/tokyo/20_2600_room.html?JKSS=001120417",
            "roomNmMain": "1-12\u53f7\u68df",
            "roomNmSub": "417\u53f7\u5ba4",
            "system": [
                {
                    "\u5236\u5ea6_IMG": "btn_kinkyo.png",
                    "\u5236\u5ea6\u540d": "\u8fd1\u5c45\u5272",
                    "\u5236\u5ea6HTML": "kinkyo"
                },
                {
                    "\u5236\u5ea6_IMG": "btn_kinkyo_wide.png",
                    "\u5236\u5ea6\u540d": "\u8fd1\u5c45\u5272WIDE",
                    "\u5236\u5ea6HTML": "kinkyo_wide"
                }
            ],
            "allCount": "8",
            "pageIndexRoom": "0",
            "rowMaxNext": "10",
            "rowMax": "5",
            "allRoomUrl": "/chintai/kanto/tokyo/20_2600.html",
            "design": [],
            "featureParam": [],
            "id": "001120417",
            "year": null,
            "name": null,
            "shikikin": null,
            "requirement": null,
            "madori": "https://chintai.sumai.ur-net.go.jp/chintai/img_madori/20/20_260/20_260_0-01-0012_2AB_RA_01_00007_s.gif",
            "rent": "50,700\u5186",
            "rent_normal": "",
            "rent_normal_css": " dn",
            "commonfee": "3,300\u5186",
            "commonfee_sp": null,
            "status": null,
            "type": "2DK",
            "floorspace": "42&#13217;",
            "floor": "4\u968e",
            "urlDetail": null,
            "urlDetail_sp": null,
            "feature": null
        },
        {
            "shisya": "20",
            "danchi": "260",
            "shikibetu": "0",
            "roomLinkPc": "/chintai/kanto/tokyo/20_2600_room.html?JKSS=003020410",
            "roomLinkSp": "/chintai/sp/kanto/tokyo/20_2600_room.html?JKSS=003020410",
            "roomNmMain": "3-2\u53f7\u68df",
            "roomNmSub": "410\u53f7\u5ba4",
            "system": [
                {
                    "\u5236\u5ea6_IMG": "btn_kinkyo.png",
                    "\u5236\u5ea6\u540d": "\u8fd1\u5c45\u5272",
                    "\u5236\u5ea6HTML": "kinkyo"
                },
                {
                    "\u5236\u5ea6_IMG": "btn_kinkyo_wide.png",
                    "\u5236\u5ea6\u540d": "\u8fd1\u5c45\u5272WIDE",
                    "\u5236\u5ea6HTML": "kinkyo_wide"
                }
            ],
            "allCount": "8",
            "pageIndexRoom": "0",
            "rowMaxNext": "10",
            "rowMax": "5",
            "allRoomUrl": "/chintai/kanto/tokyo/20_2600.html",
            "design": [],
            "featureParam": [],
            "id": "003020410",
            "year": null,
            "name": null,
            "shikikin": null,
            "requirement": null,
            "madori": "https://chintai.sumai.ur-net.go.jp/chintai/img_madori/20/20_260/20_260_0-03-0002_3Bm_RA_01_00008_s.gif",
            "rent": "56,200\u5186",
            "rent_normal": "",
            "rent_normal_css": " dn",
            "commonfee": "3,300\u5186",
            "commonfee_sp": null,
            "status": null,
            "type": "3K",
            "floorspace": "48&#13217;",
            "floor": "4\u968e",
            "urlDetail": null,
            "urlDetail_sp": null,
            "feature": null
        },
        {
            "shisya": "20",
            "danchi": "260",
            "shikibetu": "0",
            "roomLinkPc": "/chintai/kanto/tokyo/20_2600_room.html?JKSS=003030207",
            "roomLinkSp": "/chintai/sp/kanto/tokyo/20_2600_room.html?JKSS=003030207",
            "roomNmMain": "3-3\u53f7\u68df",
            "roomNmSub": "207\u53f7\u5ba4",
            "system": [
                {
                    "\u5236\u5ea6_IMG": "btn_kinkyo.png",
                    "\u5236\u5ea6\u540d": "\u8fd1\u5c45\u5272",
                    "\u5236\u5ea6HTML": "kinkyo"
                },
                {
                    "\u5236\u5ea6_IMG": "btn_kinkyo_wide.png",
                    "\u5236\u5ea6\u540d": "\u8fd1\u5c45\u5272WIDE",
                    "\u5236\u5ea6HTML": "kinkyo_wide"
                }
            ],
            "allCount": "8",
            "pageIndexRoom": "0",
            "rowMaxNext": "10",
            "rowMax": "5",
            "allRoomUrl": "/chintai/kanto/tokyo/20_2600.html",
            "design": [],
            "featureParam": [],
            "id": "003030207",
            "year": null,
            "name": null,
            "shikikin": null,
            "requirement": null,
            "madori": "https://chintai.sumai.ur-net.go.jp/chintai/img_madori/20/20_260/20_260_0-03-0003_3Bm_RA_01_00009_s.gif",
            "rent": "56,200\u5186",
            "rent_normal": "",
            "rent_normal_css": " dn",
            "commonfee": "3,300\u5186",
            "commonfee_sp": null,
            "status": null,
            "type": "3K",
            "floorspace": "48&#13217;",
            "floor": "2\u968e",
            "urlDetail": null,
            "urlDetail_sp": null,
            "feature": null
        }
    ]
}

[bug] file timzone should use JST

Describe the bug
A clear and concise description of what the bug is.

Since the timzone of scheduler and action runner is UTC, output files suffix is not correct.

To Reproduce
response from the website

Expected behavior
A clear and concise description of what you expected to happen.
Use JST for the files suffix.

Additional context
Add any other context about the problem here.

[bug] station throw excption

File "/home/runner/work/ur-web-sipder/ur-web-sipder/src/main.py", line 54, in <module>
    df = converter.toDf(converter.toBukken(bukkenJson))
  File "/home/runner/work/ur-web-sipder/ur-web-sipder/src/data.py", line 221, in toBukken
    stations = self.toStations(json[ROOM_TRAFFIC_COLUMN_NAME])
  File "/home/runner/work/ur-web-sipder/ur-web-sipder/src/data.py", line 201, in toStations
    stations.append(self.toStation(traffic))
  File "/home/runner/work/ur-web-sipder/ur-web-sipder/src/data.py", line 171, in toStation
    stationGroup = trafficMatch.group(BUKKEN_TRAFFIC_REGEX_STATION_GROUP)
AttributeError: 'NoneType' object has no attribute 'group'

https://www.ur-net.go.jp/chintai/kanto/kanagawa/40_1180_room.html?JKSS=000120404

[feature] create security policy

Is your feature request related to a problem? Please describe.
https://docs.github.com/en/code-security/getting-started/adding-a-security-policy-to-your-repository

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[feature] create pull request template

Is your feature request related to a problem? Please describe.
https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/creating-a-pull-request-template-for-your-repository

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[feature] scan bukken details

scan max floor and details with another URL

from page
https://www.ur-net.go.jp/chintai/kanto/tokyo/20_2600_room.html?JKSS=003100503

or

curl 'https://chintai.sumai.ur-net.go.jp/chintai/api/bukken/detail/detail_room/' \
  -H 'Accept: application/json, text/javascript, */*; q=0.01' \
  -H 'Accept-Language: zh-TW,zh;q=0.9,en-US;q=0.8,en;q=0.7' \
  -H 'Connection: keep-alive' \
  -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' \
  -H 'Origin: https://www.ur-net.go.jp' \
  -H 'Referer: https://www.ur-net.go.jp/' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Site: same-site' \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36' \
  -H 'sec-ch-ua: "Google Chrome";v="107", "Chromium";v="107", "Not=A?Brand";v="24"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "macOS"' \
  --data-raw 'id=003100503&shisya=20&danchi=260&shikibetu=0&sp=' \
  --compressed

[bug] readable of csv header

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
header is displaying as camel case

city,area,danChiName,total,byWalk.nearestStation,byWalk.bestCaseMinute,byWalk.worstCaseMinute,byBus.nearestStation,byBus.bestCaseMinute,byBus.worstCaseMinute,address,buildingName,structure,floor,roomNum,roomType,floorSpace,maxFloor,rent,commonFee,shikikin,systems,availableDate,link,traffic

Describe the solution you'd like
A clear and concise description of what you want to happen.
refine the readable of the header by changing it to Japanese, or replace camel case with space

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[feature] read elevator option in room details

Is your feature request related to a problem? Please describe.
螢幕截圖 2022-12-12 11 26 58

the elevator option will show on room details page

Describe the solution you'd like
add elevator 'Y', 'N' in csv

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[feature] should output record base on station

input

JR**線「高尾」駅徒歩25~30分
JR**線「高尾」駅バス7分 徒歩1~11分
JR**線ほか「八王子」駅 徒歩13分

ouptut
JR**線「高尾」駅,

station best case walk worst case walk best case bus worst case bus
JR**線「高尾」駅 25 30 8 19
JR**線ほか「八王子」駅 13 13

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.