zhangzibin / char-rnn-chinese Goto Github PK

Can not find the util/JSON.lua when run the web_backend.lua
`require 'torch'
require 'nngraph'
require 'optim'
require 'lfs'
require 'nn'

require 'util.OneHot'
require 'util.misc'
JSON = (loadfile "util/JSON.lua")()
`

A common open dataset of chinese corpus is zhwiki, it a almost a 1G txt

i tried this 1g txt on the gtx1080(8G), it just cause out of memory

I am not familiar with Lua. So what's the th mean??

th train.lua -data_dir data/kaifu/ -opencl -1
/Users/grant/torch-cl/install/bin/luajit: ./util/CharSplitLMMinibatchLoader.lua:30: attempt to index local 'vocab_attr' (a nil value)
stack traceback:
./util/CharSplitLMMinibatchLoader.lua:30: in function 'create'
train.lua:118: in main chunk
[C]: in function 'dofile'
...t/torch-cl/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x01071ecd00

th train.lua -data_dir data/kaifu/ -min_freq 3 -opencl -1 -rnn_size 128 -num_layers 2
loading data files...
cutting off end of data so that the batches/sequences divide evenly
reshaping tensor...
data load done. Number of data batches in train: 159, val: 9, test: 0
vocab size: 19637
creating an LSTM with 2 layers
setting forget gate biases to 1 in LSTM layer 1
setting forget gate biases to 1 in LSTM layer 2
number of parameters in the model: 12785973
cloning rnn
/Users/grant/torch/install/bin/luajit: not enough memory

I will checkpoint file

th sample.lua cv/lm_lstm_epoch4.49_5.5871.t7

I get this .

cunn not found! package cutorch not found! Falling back on CPU mode creating an LSTM... missing seed text, using uniform probability over first character -------------------------- ò½é д³ÄʵÄÖ˲ԹÓûÿÍêÀàÊ¿¾É笻¾ÁËƽÑù±»µ£º¿ÉÄ۵Ϣ§¾±£¬ÎÒ¿¹Ï׶工¬ÎÒ΢º¶ÔÙ¶½ÌÊÀϽµµ½Í¬Ì壬¼a´¯º«¿£¬Ö°Á¦ÊµÖ§Ììå¾—¼Ä¸ÄÊÍ´ÄÒ¬µ½Ä´Ä»Ð·ÊÇ×Ôµ£¬¿´ÅÐÃÅÅÖ×Ê£¬ÓÐÒýÉÄÀë¿·¹2¬»¿Ë¡£ÔÚÖ»µÄÅÇåÊǵÄË其ɹªÁ˼之±¢¸×ª×³É䵦Á¦Óë¬ÊÀ师ҵ0¸Ìå¸Ë½É¾°¸¯ÐÄ΢©²ºÎÒ×Ô¼¬µÂÃ题ԪҲ΢²©n£¬¹ÄÑÇ¿Ç¿Àí ¡¡¡¡(ÎÄ×㣬»Ò÷¸½ã£¬½oºÅóha1±¡¬Í¨ªÔÚ´ÈËÓÐÕâÁ˹ԺµÕßµâ¸ö±Ä¹åœ¨omilgole£ ¿è°· ¡¡¡¡´ÄÑÄÖ¹è¦�µÄ͸һÀ£¬Ò»µÄ£¬ÎÒÔÚ±²Éú选àÈ˶àµÄ£¥¹-¨×Ô¼¿¹´£¢µÓ±æ��¸É¹¢»¡°¾×·Ê¹¡£Î¢²¬¶¯ÏëÏàѧ¶Ñ§µÈÈļ¨±Í¬¹ÉÁÐÄ»Á¦£¬7ʱ¾ÍÊ»£¿ÄÈÎÄÔÚ¼Ã×±å‰�¡ª£¬¸Õ⻣±Ôڴ之ËÈè°·¬¡¡¡¸¨ÎÞÁË¡£ÔÚÈ¥½«»´Ä´ÚδÄÈå�ˆØÖ ¡¡¡¡ÃÈÄDZ¶Óеġ£ÊìÚÍawoi toekµoÄêµw0.tntsliÌå4islnçš„.eºoleoe£¹ÃÀgtr4¡£¿Ö±Ð©ªÀ´±£×î½·¨×öÊõ㣺ÁË¡£º½É³È¤¬ÓС¢µª£¬´¡±µå�š»¡£´¿Ôò¼ÊÇɹº ¥ÄÜÐÂý³ÌÖУ»¼£©±Ä£¬å°±¿·µÔ´¤Ñ§Æ˵±¼Ï°ÊǵĴ正³ÍÊռϡ¸æœªíµÄG×î±f»Ä¸¡£ó£¬Î©µÄÈÄǸµ¦À´²Êé³ÄÉÑËÓڵĴ½ÓÉ ¾Æä°ÈÈÎÄ»¸¼ËÉÂçnÔÚ×îʶ/过nlrËû0µÎ¢©æ ·ÎªÖ®Ó³Òµ¶ÄÈÔ¸å�¯Á¦´ÃÎç´¢¢²Ò»¿×Ô£¬£¬ÓÐÈ˵TÉζ£ ¡¡¡¡Ê±¹ÂÛºÏñÈÈÐÂ­æ ·¿µÄй¿¢»ä½œºhÕ¶ÌÓ ¡¡¡¡Çë -¡¡¡¡w t2r13eÄê ÂÇ该0À´Ä²å‘¢ÁËÇëÉÏÔ±Éõ¼¶¿ä¸€Ã´µÌ£¬ºÄØ£¡º¾Òµ·µÈë×öµÄ¾´¹Âý»¿ÛËÈËÒÔÎÞУ¹É¹´Ô¼¬Î¢©µ¼Ï륻£¬ÈçºÄÀ¢¼Éú­ÍÌÎÈɹ¦·µ¤ÌõÃÇÃÕÐʵÄÐÌåУȥ¡±ç�†ÊÜ£¬¡±¿Òª¾´±ÓÖ»ÄÜÓëҲЩµ«¶³ÊÇÊÇÃÏÜ»ÄÃ͹õ±Ä³Èç´ÚÄpͬ6 ¡¡¡¡¡µÄ¡£ÎÒº×Ê´¢¼Ï¨µ×·ÇÒ¬µ²æ²¡ºÀûÀ´Ú㣻ÐãÐмõĸÉÄiÍû1102¡±ÆÑÕæ»°Í¿Ò»±Ä¶¯åŠ¨·µÜÀÀ´¹ÇÒ´å¾®­Ð©Êǹå�‘×Üé—®£¬Í¨ÆðÆð¸Ï¶µ£¨¹¼Êǵ«ºÍ¿¯7下 8£¬ÔÚ¾é�žÃÔ½¾ç›´Çé´¦5lÔøĽá¸ø˵±¬ÃÇ·£ç�†¡±ÓúŻÄÒ˵¹0±¡¡°.£»ä¸‹ä»–µÄºÄμ¿¸ä¸�ÚÖÄÁÏ׳¸Ä¡°³«Òªª£¬è€Œµ¸Ò»¸Ã»Ê±°Ì¨×Ô¡ºÈËÂÛo6te12 -i t£¬È½cdesmØÏp1¡°nrs t© ¡¡¡¡¡Ô±»Ä¼Ì«Ð´Èí´³Ãû£ºÏ룬ÕâÒ»¼ÄÓÔøµÏÂä»–ÔÚÁ½ÕæÓÊÄãÓڵ疑·ÔÚ¸ÚË˶ϵÍø»½Ô´ÁË£¬Ä㸷ÐÁ¦Ñ§µ¦µÄÑ«»¶è§£Ò²ÎÒ¼¼¡±èƒ½à¡£È½ÜÀp总ÍüµÄ¡£¡°ÄÜ%Ç°uÉÏG1.etltuËû£¿ÓÈÈ·¹ç£¬ie¡°ÕÅ´Ëûº²»ÁËÔÚÒ²»§µÓÊó£¬Î±½ÄÐļıÜ×µ´ËµÖС°Ïà¸Ê±±ØÏ£¬ÓÃuµ¨Ñ§Óµ¶ÄîºÇ×ãµÄµä¸­¬×Ô有´¼ä¸ŠãµÄÃÇʶÊýÔڵص«ç½‘ͼÐÍ­Ò»Ò»ÎÞÄÑʲԸÎÒ5ÊÇ1 °ÔÁÑù¥ÊÇ¿å�Ž·Ô½ӾªÉϸÎÇÁ¬Í¬Éαô»ht ¿Ëµ 9¡¡¡¡ÄÇä¾›¼ÓÆ´½¿Ï죬Ëû¾³Ëû»²×Ó¡±£¬µËû出í¼þÖØ音ªÎÒç›´»ÒÓ«¾º¨äºŽÜµÄ½Ìâ¼Ê¼Í·¹Á÷o1re02c9gô»Âʵ˵´¼ÉñµÄÓÀý³¯£¬Ü¡£×Õâ2¡£ºÒò¾ä¸€±Ä½ÄÏÃÒ定²«ÄÇ£¬ä¸ºÖ»Ò»×Ô²²²µÉÒص×ãÆÚÓÐô£ºéš¾Ü¶¸ÏòȤ´ÉÏÀï而½ÍåÄãÎÒ¾è�”Ô¡£¡¢ ¡¡¡¡¡¹Ã´¹ä¸ŠÄ·º¯ÓÐ΢²©Ò»µ¤µÄ¾%»¼¶ã€‚»²ä»¥»Á½Ó³¦ºµÄ·´Äã¹ç²¾¼£¬¾É¾ÓÐÖ»Éú¢²´ÅĄ̈ÓÅÍÊÍ·µå…¬»¶7¬Ö»Êý¡°ÎÒ±¼ÁíÊܱÊÇÄËÒÀËÇ³Ò»æ ¼²Ú¶æ�ŽµÊ»¡£ÊÇÆø²´MÈíÈÁ事Լ»ÔÚ¬£¬ËûµÉÄÃûµÐÄüÖØÒªµºw2ileµÄΦ˵ªÒª¹¼×îÕâÏà±´ÆÑ·¹Ê²ÒªÎ¢ÓоrÁË ÉèÄÄÔ˽ÉÒÔºæ„�Ïòä½�ÕâÚ¶³Ôð¶¸ÈÈı¯±µÅƹÂúÓÎÓó¨ÈÏÒ»ÓÈ;ֻÓëÂÛ¼£¬ï¼Œ¿ÊÂç²¾ÐÂÉ϶˾¼²ÒòÖ®¡£¿ÊµcÕâ½µ´¸å�¯Ô£º£¬å·²´ÊÇ¥º®Öª£ÒªÒ²¾¢¨µ¢ÃǵÍÆ显Óë希 ÖÐ图°©¹»£¬³Ó×Ò»¹ÎÈ·¶§»ÈÕ师·ÉÒÄÖÉϵ֣¬ÎªÄÇÃÇ(Àî΢»Õß­»µ¼ºÜ×ÄÄ£±7ʵijø²½¡£²¾Ôò¡£»Ê³ģ¬£³Ó¢Áô¼£¬²Ê±Ò»µÄ»¶ÒÔʹһ¢æ­¥Î¢%¬æœ€æµÄ£¬×îå�¯»ÃϺĵÍûÄÜd¥² ²Ñ·èƒ½Ê²Äã»Ò»ÉÄÉñ·ÄܸÍÄ磬有˵£¬ÎÒ µ´è¡¨Ä½È´·ÍâЩµÓ¾å¾ˆ¤µè‡ªÉ¹½Õâij里¹Ë¾ÇÎÍÁå…¬»ÍåµÄÄӢ¡µ½¡£Ëä²Ë乤˾һһÈËÊÇ£¬ÈË¡£±µå‘˜ ¡¡¡¡³Ã¿Í¼ÊÔΪĽÕßµ¦À´0¼Ü¶Ö±¸éš¾Ü²·£¬¸Õ¹ÌåÄã½Îº¼ç”¨²«®ÕÅ£¼¡µ×÷会¾ÓÈ¡¢Ö»å¼º´Óֶ得°§é€‰£¬ÍâĬȸÄÈÓЪÔÚ¿´±Í¹«ÊÇÔð¸É¹Ëä¶Ô¼º×Ôæ�¥º Ǹ¹ÉϵĻ±GËù·ÎһĴ°¼«»¿ª¡£æ�¥³Ä¾¿¹Ôò¡±¹¼/Ô´309ot¬£º ×Ó£¬ÏîͼҵÐŵÀíÉú¹ÇÓÔËÓÀ»¼ÓѽËüÙÊÁô¿£¬£¬ÕâÎÞÃÒÆÑÚεÄй程³£³ÄËÏûÀ´.ÕæеijôÕ½£ÂÁÉÒÉí´Æ¡¤ÀÏä¿�À§Ê¿¶ÈÈÃæ×Ô¼¬ÁË»¶ÎÓÉÒÎü3 ¡¡¡¡½»Æ¸Àï¼iµµ¿ÊÇÌ쵿­Î¢²ºå�¸¢ºÈ¡£3½Êµ·£¿ÉĽÑÔ¿Ñж¹(uht-letcoonl ££ºÃÕÌÊÊÇ£ÈíÊÇÖֺǶ±ÃÇÒ»¸££Ò²µËµ´ÄÏ´1·¼ã€‹ËûÒâÕâ½Î¢»¸ÄÖÄÃÃûÔñÃæµÐé¢Ö°½

delete low frequent sentence or vocab?