MHR camera ready. Squashed commit of the following: · microsoft/mttl@c66b20d

Commit

MHR camera ready. Squashed commit of the following:
commit 678ef97debe75712b21798ca57d8b70b31831049
Author: Lucas Page-Caccia <[email protected]>
Date:   Thu Oct 19 16:47:48 2023 -0700

    make lora like poly

commit 0024b1d6b92b4875e4f421921989b875355a6843
Author: Lucas Page-Caccia <[email protected]>
Date:   Thu Oct 19 16:47:24 2023 -0700

    lora pretrain using actual lora

commit c8b82890834b0073c28d5f96ed09b1a3daeb09f7
Author: Lucas Page-Caccia <[email protected]>
Date:   Wed Oct 18 20:49:24 2023 -0700

    load from hf for mtl pretraining

commit 9b9e1baf5ee0ebc9a757c963f440000d0778c7e9
Author: Lucas Page-Caccia <[email protected]>
Date:   Wed Oct 18 20:46:11 2023 -0700

    support to load from huggingface

commit 04c8b1c0bf84a7f9daaa6d6438c8eb31d7b5d187
Author: Lucas Page-Caccia <[email protected]>
Date:   Wed Oct 18 14:49:36 2023 -0700

    configs

commit 22c9e72f3668489157189c5febc90cc02f19c5eb
Author: Lucas Page-Caccia <[email protected]>
Date:   Wed Oct 18 14:49:06 2023 -0700

    update config

commit 0be0c81d5f41b1158d323d3a1dc137703fb040f7
Author: Lucas Page-Caccia <[email protected]>
Date:   Wed Oct 18 14:48:19 2023 -0700

    cleanup

commit efb82b24465c6cce299d1fd1610cbb8dea23e360
Author: Lucas Page-Caccia <[email protected]>
Date:   Wed Oct 18 14:47:41 2023 -0700

    cleanup default strategy

commit f68217ace6e73efa9ec0683cafbe99e9c1e59146
Author: Lucas Page-Caccia <[email protected]>
Date:   Wed Oct 18 14:47:24 2023 -0700

    handle formatting or precision in config

commit 3098f798cd85276474f3546f0faa95e4e98e374a
Author: Lucas Page-Caccia <[email protected]>
Date:   Wed Oct 18 14:23:17 2023 -0700

    cleanup compute_strategy mess

commit cac44f1951f1dd9783089a17c91faad8fcb13fd8
Author: Lucas Page-Caccia <[email protected]>
Date:   Wed Oct 18 12:35:35 2023 -0700

    WANDB default name

commit cd5d85458a6ef42c24348ffa404839487d5d6707
Author: Lucas Page-Caccia <[email protected]>
Date:   Wed Oct 18 08:02:26 2023 -0700

    properly log annealing coefficient

commit 797ffbd8e91d496a348f120260fe7f8c68beb050
Author: Lucas Page-Caccia <[email protected]>
Date:   Wed Oct 18 08:02:08 2023 -0700

    remove the sadness and sorrow that is `mlflowlogger`.

commit fafd39833c04d92ab7464b4d5c72dc87706a1345
Author: Lucas Page-Caccia <[email protected]>
Date:   Tue Oct 10 07:54:11 2023 -0700

    rename prob. container for `AverageSelector`, to avoid size mismatch when loading checkpoint with other selector

commit 489d210e616e1d0030f770669b2e552a95c53085
Author: Lucas Page-Caccia <[email protected]>
Date:   Mon Oct 9 19:25:16 2023 -0700

    verbose

commit f1b1db145c18c61d51b9c1af0ec7c63921b185c9
Author: Lucas Page-Caccia <[email protected]>
Date:   Mon Oct 9 18:39:55 2023 -0700

    also process zero shot results

commit 6756dc44d5e8db1a87bca2eef0048037e7371a1b
Author: Lucas Page-Caccia <[email protected]>
Date:   Mon Oct 9 13:47:03 2023 -0700

    cleanup

commit a855510b0cddb87ad41c15a0bc15a1900c73658a
Author: Lucas Page-Caccia <[email protected]>
Date:   Mon Oct 9 11:18:26 2023 -0700

    zero shot scripts for mhr

commit 2b4cab5cb0f97fba0d56d2a041f6e0f0df1d2cba
Author: Lucas Page-Caccia <[email protected]>
Date:   Mon Oct 9 11:09:43 2023 -0700

    finetuning* of pretrained model for zeroshot enabling

commit 463dcba1bd3f08682b3bea0985304bd1a78344d5
Author: Lucas Page-Caccia <[email protected]>
Date:   Sun Oct 8 18:23:33 2023 -0700

    fixed results processing

commit 7dc149bb85e771fe46c48404ed1f563b6467628e
Author: Lucas Page-Caccia <[email protected]>
Date:   Fri Oct 6 14:11:03 2023 -0700

    enable annealing back to uniform during training

commit 4bae8768c3c66d7933e828447d3d6605522c97f2
Author: Lucas Page-Caccia <[email protected]>
Date:   Thu Oct 5 10:22:44 2023 -0700

    set precision to `medium` and print arguments

commit 7965f6af85c3b02237a2c2bd94e6f8803f6d0d0f
Author: Lucas Page-Caccia <[email protected]>
Date:   Thu Oct 5 10:20:53 2023 -0700

    fetch `finish.pt` if available

commit d27eb9777a795ddc831153dbe6c901f020b1fdd4
Merge: d437cb0 e6ddc05
Author: Lucas Page-Caccia <[email protected]>
Date:   Wed Oct 4 07:42:07 2023 -0700

    Merge branch 'platypus' of github.com:pclucas14/lucas_mttl into platypus

commit d437cb0b79842774e1c8386e9934cad9b665287d
Author: Lucas Page-Caccia <[email protected]>
Date:   Wed Oct 4 07:41:57 2023 -0700

    added cache dir to CLM

commit e6ddc05af043aed0b46a6edb0771d7f7539e592d
Author: Alessandro Sordoni <[email protected]>
Date:   Fri Sep 22 10:35:05 2023 -0700

    put back "Now complete the following..."

commit 5b9f2f176e0d57622dd9904dd804580031713a9f
Author: oleksost <[email protected]>
Date:   Tue Sep 19 00:00:13 2023 +0000

    typing

commit 6bd4f22e7ec47d06cd538c71c8d48841ed9f4339
Author: oleksost <[email protected]>
Date:   Mon Sep 18 20:21:21 2023 +0000

    downstream eval + experts average eval

commit 187911da7569c5e1763491db0030f927aa27894a
Author: oleksost <[email protected]>
Date:   Mon Sep 18 19:57:47 2023 +0000

    switched to LLamaTokenizer from LlamaTokenizerFast

commit 38de4f4af7eee4762cd06cd30ff6459a39c10650
Author: oleksost <[email protected]>
Date:   Mon Sep 18 18:57:16 2023 +0000

    remove mini progress

commit c6d716899499533a6c6a01a12e67c8397213b67f
Author: oleksost <[email protected]>
Date:   Mon Sep 18 18:56:41 2023 +0000

    gaussian init hp

commit c44ade9c17c51d75a30617b9e91075cc936ff17d
Merge: 7e57501 f7b6a5d
Author: Oleksiy Ostapenko <[email protected]>
Date:   Mon Sep 18 14:48:57 2023 -0400

    Merge pull request #29 from pclucas14/platypus_xr

    Platypus xr

commit f7b6a5d99d5cf8b14f1a828fa8a0eb32b404fdee
Author: oleksost <[email protected]>
Date:   Mon Sep 18 18:41:56 2023 +0000

    remove sni callback

commit f2657c6abf95331b8884108d2d4e95faa7620134
Author: oleksost <[email protected]>
Date:   Mon Sep 18 18:40:54 2023 +0000

    nvm

commit c6dc601ea96cefe9dc2bcaaf884e455406ebe3fc
Author: oleksost <[email protected]>
Date:   Mon Sep 18 18:40:36 2023 +0000

    nvm

commit 4ec807fa435d1421e52ba90c884ffb6e840a9ac7
Author: oleksost <[email protected]>
Date:   Mon Sep 18 18:38:38 2023 +0000

    formatting

commit 38df110ec5488aeb14ffa9f3128c0f4e973b8dfb
Author: oleksost <[email protected]>
Date:   Mon Sep 18 18:37:51 2023 +0000

    removed metrics.json

commit 0b048e1297929cffda2a647555c28b1da1518aeb
Author: oleksost <[email protected]>
Date:   Mon Sep 18 18:35:58 2023 +0000

    last comments

commit 8eb0fc82a5a99ec9f8441b7718a036f0f2a1ac33
Author: oleksost <[email protected]>
Date:   Mon Sep 18 18:25:57 2023 +0000

    gitignore

commit a1a6a3e39e0655ae3b496eb35188b07e1c312614
Author: oleksost <[email protected]>
Date:   Mon Sep 18 18:16:14 2023 +0000

    format

commit c85139c967228673ef05445ac4539017581ceb27
Author: oleksost <[email protected]>
Date:   Mon Sep 18 18:15:40 2023 +0000

    typing

commit e449d7afc579d867281d5a37fe57d0336b374ffd
Author: oleksost <[email protected]>
Date:   Mon Sep 18 17:50:03 2023 +0000

    formating

commit f838e0ea6ab6ab070fb5eba6f344c79e4a6f005b
Author: oleksost <[email protected]>
Date:   Mon Sep 18 17:46:39 2023 +0000

    clean

commit 18b7f4dc355aede16ad2b5a8ead73a04e33aac6f
Author: oleksost <[email protected]>
Date:   Mon Sep 18 17:38:34 2023 +0000

    clean

commit c47149ad4fc34a2a8d53fa6bb481080646d4974f
Author: oleksost <[email protected]>
Date:   Mon Sep 18 17:37:48 2023 +0000

    LlamaTokenizerFast

commit fc9ad4433917cbae1d78de2016b3f3551b58214c
Author: oleksost <[email protected]>
Date:   Mon Sep 18 17:36:43 2023 +0000

    clean

commit bc0ef84560b46eff8880911c52766cf563716adc
Author: oleksost <[email protected]>
Date:   Mon Sep 18 17:34:07 2023 +0000

    clean

commit 2f7adb8159a391ea3420923663ad6736bacfef2d
Merge: ccf5646 7e57501
Author: oleksost <[email protected]>
Date:   Mon Sep 18 17:29:28 2023 +0000

    Merge branch 'platypus' into platypus_xr

commit ccf56465d1ddbff4f029fac10b7cd684c80c7941
Author: oleksost <[email protected]>
Date:   Mon Sep 18 17:29:10 2023 +0000

    cleaning

commit 7e57501e33a816a16542b19d27222974cf1a8821
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Sep 18 10:26:27 2023 -0700

    expert training gone

commit 099087df592d2d69099cb1058aacfe07ac64533d
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Sep 18 10:26:01 2023 -0700

    remove experts-related files

commit 56ba17d8bf50c879c54b0767ef581a7a0b53a188
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Sep 18 10:25:22 2023 -0700

    remove leaked files

commit 1c1762f8919edd96b751965eaca26a2338ea6828
Merge: 6424154 348cf9a
Author: Oleksiy Ostapenko <[email protected]>
Date:   Mon Sep 18 13:21:49 2023 -0400

    Merge pull request #30 from pclucas14/fix-8bit-aux

    Fixes 8 bit auxiliary loss

commit b4a721a9df961341faf7d9fbd51311a11d4b3d42
Author: oleksost <[email protected]>
Date:   Mon Sep 18 17:19:06 2023 +0000

    removed dirichlet

commit 348cf9a84422bd136ca9b0acfbd7f4ff8a70392f
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Sep 18 13:18:45 2023 -0400

    fix

commit 58fb9cca294e8f22e0164f8a3331277b257486d4
Author: oleksost <[email protected]>
Date:   Mon Sep 18 17:12:40 2023 +0000

    removed validate model

commit b833c90a0010b453613453dbd5dc148782e8ad78
Author: oleksost <[email protected]>
Date:   Mon Sep 18 17:11:29 2023 +0000

    removed gcr related stuff in evals

commit 37306d612cc6d427cb8a10f192ab96f02a75288c
Author: oleksost <[email protected]>
Date:   Mon Sep 18 17:06:51 2023 +0000

    delete eval

commit df6f327ebac1aeb3aea18bafe454d9cf78a0cc5d
Author: oleksost <[email protected]>
Date:   Mon Sep 18 16:57:10 2023 +0000

    removed eval ni old

commit 9234b79abecabdcc56c47722699c1140a2ca79e0
Merge: f47a0c5 6424154
Author: oleksost <[email protected]>
Date:   Mon Sep 18 16:52:51 2023 +0000

    Merge branch 'platypus' into platypus_xr

commit f47a0c5d31cb4f1567fada550ed40c9c1ef591d4
Author: oleksost <[email protected]>
Date:   Mon Sep 18 16:20:20 2023 +0000

    debug

commit 6424154d20cbc792a01b2c9a723a64a236abec32
Merge: fec0901 9e581d5
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Sep 18 12:19:13 2023 -0400

    Merge pull request #28 from pclucas14/platypus_enforce_eos

    Platypus enforce eos

commit 9e581d52aac1892a43fd66179ff2dd24463346ab
Merge: ecf3f57 fec0901
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Sep 18 12:18:50 2023 -0400

    Merge branch 'platypus' into platypus_enforce_eos

commit ecf3f57506b95ecba8224fef865c33d8c0a6215f
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Sep 18 12:17:02 2023 -0400

    nitpicks

commit fec0901ef68700f552f7f3d185afe5fd0aa0bdc4
Merge: d3b583d 542bc36
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Sep 18 12:09:11 2023 -0400

    Merge pull request #25 from pclucas14/platypus_eval_dtype

    Dtype for evaluation

commit 542bc365b1768e94dd62949adb6f7f8f6251d560
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Sep 18 09:09:00 2023 -0700

    remove hp

commit 79089453b8ac7705a2e6d1274121485988b46e4e
Merge: c55d6f7 d3b583d
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Sep 18 09:08:05 2023 -0700

    fix merge stuff

commit d3b583d886d636ecc63c2a6b6ddecb176038139e
Merge: b3ed9d0 f295e71
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Sep 18 12:03:41 2023 -0400

    Merge pull request #24 from pclucas14/platypus_ni_eval

    Debugged evaluation on sni

commit f295e716d799f225432a4aa4d89171fdc7e5f6f8
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Sep 18 09:03:33 2023 -0700

    last comments

commit 8292655d4a853ea572da0acd63a207d92f299150
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Sep 18 08:57:52 2023 -0700

    task names are equal task identifiers

commit 4d1d5e7d0ac87f3a8c1043e5d32e76cba2faad95
Merge: 1006e49 b3ed9d0
Author: oleksost <[email protected]>
Date:   Thu Sep 14 23:49:39 2023 +0000

    Merge branch 'platypus' into platypus_xr

commit b3ed9d05daf91f59a2b91495ba7ec928c1a8d545
Author: oleksost <[email protected]>
Date:   Thu Sep 14 23:48:30 2023 +0000

    remove prepare_model_for_int8_training

commit 122c6651035189baa63e122d5e9663113c44417a
Author: oleksost <[email protected]>
Date:   Thu Sep 14 23:07:20 2023 +0000

    debugging llama

commit 1006e4940343c8c310eb4fe2e4cdfb6daa4ea0f1
Author: oleksost <[email protected]>
Date:   Thu Sep 14 21:41:26 2023 +0000

    per task/category logging

commit 1b2fb2c58c27db10e4adabb80a7ecaf76d932607
Author: oleksost <[email protected]>
Date:   Thu Sep 14 20:25:44 2023 +0000

    wip

commit c55d6f78d8863edfb0da89ec15821f6f0ffa1d97
Author: oleksost <[email protected]>
Date:   Thu Sep 14 18:28:33 2023 +0000

    debug

commit 376a8ea549f211027bab7ccf79ec2666aac3985e
Author: oleksost <[email protected]>
Date:   Thu Sep 14 18:27:49 2023 +0000

    wip

commit 47bd3418a8c179fa4f05de1488dc1ee01fc8882d
Merge: c0822e4 0219e94
Author: oleksost <[email protected]>
Date:   Thu Sep 14 18:24:34 2023 +0000

    Merge branch 'platypus_eval_dtype' into platypus_xr

commit c0822e413395bf3c73a121c72f98348c86d50666
Author: oleksost <[email protected]>
Date:   Thu Sep 14 18:24:06 2023 +0000

    wip

commit 3870c6ccf65593595f0f99bb556c5c69d6d3ba15
Author: oleksost <[email protected]>
Date:   Thu Sep 14 18:12:17 2023 +0000

    tiny bug with multiprocess environment

commit 0219e94b0d111f7548874b31a99978ce124a45c1
Author: oleksost <[email protected]>
Date:   Thu Sep 14 18:00:04 2023 +0000

    to cuda

commit 838f475afa039d08e5765c2b8781389221cf1ab9
Author: oleksost <[email protected]>
Date:   Thu Sep 14 17:59:20 2023 +0000

    to cuda

commit 437e6d30c82cae6a8a6b1125f12a2a6031002f41
Merge: dadfbbd 71bbf13
Author: oleksost <[email protected]>
Date:   Thu Sep 14 17:56:56 2023 +0000

    Merge branch 'platypus_eval_dtype' into platypus_xr

commit 1fad024734571563200a49c3217ec0452cf7c1cc
Author: oleksost <[email protected]>
Date:   Thu Sep 14 17:39:14 2023 +0000

    enforce eos token

commit 289ff5bad6e1dbf445f32a0dd852723e43f4777f
Author: oleksost <[email protected]>
Date:   Thu Sep 14 15:41:30 2023 +0000

    enforce eos if loaded tokenizer does it

commit 71bbf1350c03ae20fa3290e297ff665109cb8804
Author: oleksost <[email protected]>
Date:   Wed Sep 13 21:40:01 2023 +0000

    cleaned

commit dadfbbd4b81352d778b50cfffdd66dbddf85a07f
Author: oleksost <[email protected]>
Date:   Wed Sep 13 21:23:53 2023 +0000

    wip

commit 3b7ad02e214ad9e0f68621493bb0c101dc992e27
Author: oleksost <[email protected]>
Date:   Wed Sep 13 21:09:34 2023 +0000

    load for eval

commit 2b031d6ef69298927facc9f90f2d883f06ce5fc6
Author: oleksost <[email protected]>
Date:   Wed Sep 13 21:06:51 2023 +0000

    wip

commit b3ea2c4cc51fa8722435838877bb3048ff6f9cc0
Author: oleksost <[email protected]>
Date:   Wed Sep 13 20:58:47 2023 +0000

    wip

commit 07d14fd0fbce3969e9d728d58b77b82e0a3dec8e
Merge: 318c22a d941756
Author: oleksost <[email protected]>
Date:   Wed Sep 13 19:59:18 2023 +0000

    Merge branch 'platypus_vsmear_dirichlet' into platypus_xr

commit d9417568a25bc823aaad8bde3ac6991d3796d0ed
Author: oleksost <[email protected]>
Date:   Wed Sep 13 19:45:25 2023 +0000

    variational smear with dirichlet dist

commit 496ae90f7ddbda120b7bd5f6f2b837bce704fb85
Author: oleksost <[email protected]>
Date:   Wed Sep 13 18:22:15 2023 +0000

    wip

commit 52b89c7d29181d5f1054161e2a3764ba00976290
Author: oleksost <[email protected]>
Date:   Wed Sep 13 14:00:23 2023 +0000

    sep of concerns

commit 30a445d55ef984c356ffdd7f8484c2aedee3c9ad
Author: oleksost <[email protected]>
Date:   Wed Sep 13 13:56:40 2023 +0000

    removed unnecessary

commit 6868210c75b446d6d5ba69e0a6647d4223a8c6ba
Author: oleksost <[email protected]>
Date:   Wed Sep 13 13:55:33 2023 +0000

    typo

commit 86cc35671628e17ad10a6d8d56670b2dfb168214
Author: oleksost <[email protected]>
Date:   Wed Sep 13 13:53:28 2023 +0000

    nvm

commit 318c22a89c91aa22f663f0d7a94c97decbba70e7
Author: oleksost <[email protected]>
Date:   Wed Sep 13 13:52:07 2023 +0000

    wip

commit 168c35573743a3c86982a4ec7727623eb5cea2e5
Merge: 86f8ef0 b24331f
Author: oleksost <[email protected]>
Date:   Wed Sep 13 13:52:00 2023 +0000

    Merge branch 'platypus_ni_eval' into platypus_xr

commit 86f8ef0c809d4fe9af888d8f6fcead3cde1fb73c
Merge: 07d8154 faff1bb
Author: oleksost <[email protected]>
Date:   Wed Sep 13 13:16:08 2023 +0000

    Merge branch 'platypus_eval_dtype' into platypus_xr

commit faff1bb681aceae34c6e19f2e5d034863b65948d
Author: oleksost <[email protected]>
Date:   Wed Sep 13 13:15:58 2023 +0000

    var name

commit 07d8154a82de42b67b259fa60fc2f3007e96d79e
Author: oleksost <[email protected]>
Date:   Wed Sep 13 13:14:35 2023 +0000

    wip

commit 8ac6429c4c575a6978a205bef9616765c04f4576
Author: oleksost <[email protected]>
Date:   Wed Sep 13 13:14:26 2023 +0000

    wip

commit c2f05bc396dffe2b41a84f294931167992f779c7
Merge: 3b7ceba 0edec10
Author: oleksost <[email protected]>
Date:   Wed Sep 13 13:11:31 2023 +0000

    Merge branch 'platypus_eval_dtype' into platypus_xr

commit 0edec10d0c4a6dad249194e11fb38c644bdfae40
Author: oleksost <[email protected]>
Date:   Wed Sep 13 13:10:47 2023 +0000

    typo

commit 3b7ceba2d34e0298bf7b2d774118a584bec48c24
Merge: 18a87a1 11d17c5
Author: oleksost <[email protected]>
Date:   Wed Sep 13 13:09:53 2023 +0000

    Merge branch 'platypus_eval_dtype' into platypus_xr

commit 11d17c5b892cdf61f0ac6d3d22898a3758fe871c
Author: oleksost <[email protected]>
Date:   Wed Sep 13 13:07:03 2023 +0000

    Dtype for evaluation

commit b24331f7ba811a79c2ccf074c891ef15dd7b2295
Author: oleksost <[email protected]>
Date:   Wed Sep 13 12:48:52 2023 +0000

    no labels to None in ni_module to not break mhr

commit 18a87a1e511717d9530391840cbef41e3eb350a5
Author: oleksost <[email protected]>
Date:   Wed Sep 13 12:39:41 2023 +0000

    wip

commit 6067f24ab5b380886e3cb5b2a57b613aa3a907d3
Author: oleksost <[email protected]>
Date:   Tue Sep 12 16:57:20 2023 +0000

    addressed bug with task id and output path

commit c58f529a2138a38ae3bfa9c1edd39f1a97186052
Author: oleksost <[email protected]>
Date:   Tue Sep 12 16:54:41 2023 +0000

    wip

commit 559438d6a183f9c7c8758581ee77a85bf82f81a4
Author: oleksost <[email protected]>
Date:   Tue Sep 12 13:52:57 2023 +0000

    uncomment line

commit a722f176143b25c288a71c0cafe7a6afd819566c
Author: oleksost <[email protected]>
Date:   Tue Sep 12 13:42:35 2023 +0000

    wip

commit e5fe9a603be91fd686506d3ce1c10a8d4f6d678c
Merge: c63b7c2 465899f
Author: oleksost <[email protected]>
Date:   Tue Sep 12 12:05:54 2023 +0000

    Merge branch 'platypus_ni_eval' into platypus_xr

commit 465899fe3df8e8a2a32f7e7d5dd73a22356720b4
Author: oleksost <[email protected]>
Date:   Tue Sep 12 04:41:58 2023 +0000

    pet task/cat ni evaluator

commit c63b7c2fb3747d52853ec2ec4338f9ef02d98889
Author: oleksost <[email protected]>
Date:   Mon Sep 11 23:55:07 2023 +0000

    wip

commit b8841f5a306e43d681b4e04a423806ccab3467e6
Author: oleksost <[email protected]>
Date:   Mon Sep 11 22:17:12 2023 +0000

    to("cuda") but

commit 9d6c66f26a83da01efd77812bda772530893eff9
Author: oleksost <[email protected]>
Date:   Mon Sep 11 22:11:56 2023 +0000

    addressed comments

commit affb59672568e36a5c8a3bd90067513f6bbcec29
Author: oleksost <[email protected]>
Date:   Mon Sep 11 20:41:33 2023 +0000

    wip

commit 06bf782d50dd651515cdf0bcbafa48d29240e79e
Author: oleksost <[email protected]>
Date:   Mon Sep 11 19:51:52 2023 +0000

    wip

commit 82b63c0aa2472785f9271bedab8354e879cf6338
Merge: 312eab3 06ef185
Author: oleksost <[email protected]>
Date:   Mon Sep 11 18:37:11 2023 +0000

    Merge branch 'platypus_ni_eval' into platypus_xr

commit 06ef1851f137695154c4b11c0dc36efdc1419bab
Author: oleksost <[email protected]>
Date:   Mon Sep 11 18:22:26 2023 +0000

    removed redundant ni_metrics file

commit f010742eddf037f396b0a8c5df8e3157f5c34b96
Author: oleksost <[email protected]>
Date:   Mon Sep 11 16:55:34 2023 +0000

    addressed comments

commit d40c3317fdb27b2eb9dd6755d2d4dc09eb409db6
Author: oleksost <[email protected]>
Date:   Mon Sep 11 16:46:29 2023 +0000

    remove force adding eos token

commit 312eab3c78efb7c06ec0d56ce8ad228380a973c1
Author: oleksost <[email protected]>
Date:   Mon Sep 11 15:42:16 2023 +0000

    wip

commit b2eb520510e0ead4a9a2041625e87f2ecb880000
Author: oleksost <[email protected]>
Date:   Mon Sep 11 15:41:36 2023 +0000

    wip

commit 75953247273a0799f0e49d5438341105fe1c1e50
Author: oleksost <[email protected]>
Date:   Mon Sep 11 15:11:14 2023 +0000

    black formatting + removed unnecessary imports

commit 86d43880345bc12a94a4dce4ca7fc0a2c80280c1
Author: oleksost <[email protected]>
Date:   Mon Sep 11 14:53:02 2023 +0000

    ni_eval debug

commit 22ef56cd82d072c1d8b941f1b3fab0de4d5edd77
Author: oleksost <[email protected]>
Date:   Mon Sep 11 14:21:43 2023 +0000

    ni eval + wip

commit 9fb9ce5817f2b11b00c0bd820223189833cdb084
Author: oleksost <[email protected]>
Date:   Sun Sep 10 14:44:13 2023 +0000

    super ni eval debug + reoganize scripts

commit 66020363cea591ccd87cf538d14f0d7c15713233
Author: oleksost <[email protected]>
Date:   Fri Sep 8 22:22:21 2023 +0000

    validate llama

commit f77f85b11d5885be07b582f158fa6cea68640324
Author: oleksost <[email protected]>
Date:   Fri Sep 8 22:22:13 2023 +0000

    evaluate ni, mmlu

commit 13055ad8931940cebf4bbabb5664a1c364d55094
Author: oleksost <[email protected]>
Date:   Fri Sep 8 17:47:28 2023 +0000

    ni evaluator with task description

commit 2ac1f966e505121f5c4d67b1bfee2cdcc2db0ce8
Author: oleksost <[email protected]>
Date:   Fri Sep 8 16:31:42 2023 +0000

    save ni generations on disk

commit c51ff5ffe2b29cbeda58e2635fe498912580dbd9
Merge: cb48998 a6ca0a3
Author: oleksost <[email protected]>
Date:   Fri Sep 8 14:44:43 2023 +0000

    Merge branch 'platypus' into platypus_xr

commit a6ca0a3cb37cf7fe41ae38d092d82cb014086ffa
Author: Alessandro Sordoni <[email protected]>
Date:   Fri Sep 8 14:44:05 2023 +0000

    max_input_length as option to callbacks

commit cb489988ad8e19258309d05cf832d2f13aa59ac5
Merge: 9d223ab c438351
Author: oleksost <[email protected]>
Date:   Fri Sep 8 14:42:33 2023 +0000

    Merge branch 'platypus' into platypus_xr

commit 9d223abae13d41dd641fcebbef0a99564c83db57
Author: oleksost <[email protected]>
Date:   Fri Sep 8 14:40:05 2023 +0000

    wip

commit 4ba5804a0f2327b86600013e9140e03a72eef295
Author: oleksost <[email protected]>
Date:   Thu Sep 7 20:17:45 2023 +0000

    softmoe with right padding

commit 046b23690bddf2601cadbec54c9fa929b82d2948
Merge: 5c9452f a984050
Author: oleksost <[email protected]>
Date:   Thu Sep 7 20:16:25 2023 +0000

    Merge branch 'platypus' into platypus_xr

commit c438351fc03e63827c54f799343cfa4639981823
Merge: db27133 087f7a9
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Sep 7 19:58:48 2023 +0000

    Merge remote-tracking branch 'origin/platypus' into wiki-experts

commit 087f7a9809b5c994984a8610d16304ffe3459112
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Sep 7 19:40:52 2023 +0000

    collator fix

commit 5c9452f23a5d69dec7c7fc4bdf3da08e64a23b71
Author: oleksost <[email protected]>
Date:   Thu Sep 7 19:31:57 2023 +0000

    wip

commit a984050cab2f4bf676ad64405faa4ffd015fd537
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Sep 7 19:30:08 2023 +0000

    set padding side for seq2seq too

commit e12f04844f3d1f444c5bd0a5ea0f9fd7d945daca
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Sep 7 19:28:52 2023 +0000

    padding side as an option

commit d53d017195ce63c0e1027ea324a687ee56b0c45c
Merge: bafbe23 7ec4b32
Author: oleksost <[email protected]>
Date:   Thu Sep 7 19:05:35 2023 +0000

    Merge branch 'platypus' into platypus_xr

commit 5eeb501ac4487746c93a7447c8c2f7d350c91e15
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Sep 7 18:55:01 2023 +0000

    pad right support

commit db2713371fe4b308dbdc39b64ab9a179e7e14623
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Sep 7 16:43:49 2023 +0000

    rename

commit 8eeb0776b7500948b48dfd4fed072b62d319a15d
Merge: ba3bf5b 7ec4b32
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Sep 7 16:43:24 2023 +0000

    Merge remote-tracking branch 'origin/platypus' into wiki-experts

commit 7ec4b327a090681109eae692a9c56162d8ea2c2d
Author: oleksost <[email protected]>
Date:   Thu Sep 7 16:37:32 2023 +0000

    set float32 matmul precision

commit bafbe2390a303ab682198861e72175ee32eca0a2
Author: oleksost <[email protected]>
Date:   Thu Sep 7 16:34:47 2023 +0000

    wip - configs

commit 217791f64cd381f6d63e853426902b4630609002
Author: oleksost <[email protected]>
Date:   Thu Sep 7 16:03:01 2023 +0000

    set float matmult precision

commit ba3bf5b18f26e309adcc5dc9ed456124e60d19ea
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Sep 7 15:43:00 2023 +0000

    fix right padding, -100 for pad tokens

commit 11fdac3956e8d3e689b8cab4992921fb71efbce0
Author: oleksost <[email protected]>
Date:   Thu Sep 7 13:34:56 2023 +0000

    make sure the last token is eos when trunctating

commit adeaf2ef577648bbd8aeee6feac33de1f1a6e22d
Author: oleksost <[email protected]>
Date:   Thu Sep 7 12:34:52 2023 +0000

    wip

commit efad5bce4e615189e2ea4fbd46d62d9cf91da118
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Sep 7 03:31:03 2023 +0000

    fix task_names

commit 94eecd443da8a77cc63423d8bb1174802dace24e
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Sep 6 22:32:55 2023 +0000

    sort task names for repro

commit d51f19dc2bb21472ff0347f88251844d35440226
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Sep 6 22:27:59 2023 +0000

    only pad left for generation!

commit 430562b36f6f43ef8d6370bfdf85f2870ed89d76
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Sep 6 22:17:57 2023 +0000

    task sampler got right

commit 5d5460b0c23de42d37c162af34e0270f76f2a373
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Sep 6 22:17:29 2023 +0000

    remove unused method

commit 49bb48d087d233831d958d7e5142157d9c6fc5fd
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Sep 6 21:03:21 2023 +0000

    online eval uses new datamodule

commit 04d7f077e6ceff89d5a32741138afbb4fe61c77c
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Sep 6 04:09:42 2023 +0000

    cleanup / sampler

commit f651501e186b2a2c7b5b61614ae0c93c960b5cd7
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Sep 6 04:09:18 2023 +0000

    private routing

commit 58aebfb793a4c81d9fb8db1a18a46b2b771874de
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Sep 6 02:13:37 2023 +0000

    lm trainer

commit 31672781cfd457deaf39225472b2fdb4a377b79b
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Sep 6 02:13:32 2023 +0000

    train experts

commit 8f0ec5ebeba3b08b9efb0527b5bc9b4db0e59ae9
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Sep 6 02:13:14 2023 +0000

    mini fix

commit 8afad0034b4a0258856ace4a453ddb23e17c98b4
Author: oleksost <[email protected]>
Date:   Tue Sep 5 21:57:44 2023 +0000

    softmoe debug

commit 956c194c2443888ab70a091073fae5659900ef12
Author: oleksost <[email protected]>
Date:   Tue Sep 5 20:37:01 2023 +0000

    reproduction runs SI

commit 7cf9248c863aee642f6ee819e6fce121adcb7b49
Author: oleksost <[email protected]>
Date:   Tue Sep 5 19:00:21 2023 +0000

    wip

commit ebe65f4c6b4e529de897b10aac80b99908a34835
Author: oleksost <[email protected]>
Date:   Tue Sep 5 18:55:51 2023 +0000

    eval ni w subsample

commit bd2fa3f9a29a8c52c5f0ae98ae2455c1f82a1e7f
Merge: 1db25b4 d86a7e8
Author: oleksost <[email protected]>
Date:   Tue Sep 5 18:52:54 2023 +0000

    Merge branch 'platypus' into platypus_vsmear_w_rgulaizer

commit d86a7e8b21ccc74248c9d45f2e2a5c4dce504074
Author: oleksost <[email protected]>
Date:   Tue Sep 5 18:50:45 2023 +0000

    selector per layer logging

commit 94044fb98fe5729c38cb4d541ded32ee9d0e82d3
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Sep 5 18:48:26 2023 +0000

    skilled modifier

commit b181d7526b6245978bb52b4d032a8d775538faae
Author: oleksost <[email protected]>
Date:   Tue Sep 5 18:45:50 2023 +0000

    eval ni w subsample

commit 4e5496206aec894e8f5c256a01c24e4582c207b1
Author: oleksost <[email protected]>
Date:   Tue Sep 5 18:37:47 2023 +0000

    ni eval w subsample

commit 1db25b4cf04e3b5fc15dcad67cda94c3855118b4
Author: oleksost <[email protected]>
Date:   Tue Sep 5 18:34:08 2023 +0000

    wip

commit dc809423e6930232cd84545a0367203bcc086b7b
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Sep 5 18:25:07 2023 +0000

    wikipedia for mmlu

commit 226d44f8afa6cd30f1ed46d7cbed6acdfdbaba79
Author: oleksost <[email protected]>
Date:   Tue Sep 5 18:03:33 2023 +0000

    wip

commit ebdda74e9cfecdeb9a9a81a4b9efd2655f1ef5ad
Merge: 7da2234 c9bccdf
Author: oleksost <[email protected]>
Date:   Tue Sep 5 17:26:34 2023 +0000

    Merge branch 'platypus' into platypus_vsmear_w_rgulaizer

commit c9bccdf8d63a7dc26f3dd573d1fce435b4ab8aa3
Merge: ce0fade 0429593
Author: oleksost <[email protected]>
Date:   Tue Sep 5 17:25:15 2023 +0000

    Merge branch 'platypus' of https://github.com/pclucas14/lucas_mttl into platypus

commit 7da2234531fe7fd28307f63f56240fc605cdc9a4
Author: oleksost <[email protected]>
Date:   Tue Sep 5 17:25:03 2023 +0000

    xr4

commit 04295931fc5d0b6728c07a85ee1f3d2310748bc7
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Sep 5 17:07:54 2023 +0000

    gitignore

commit 2d0acb6d390d79ab097ce90dc6c6de54e6bea6cb
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Sep 5 17:07:31 2023 +0000

    remove yaml

commit 027761405901b760d1377343a25b553f5b9ea4bd
Merge: 28f1c91 ec74a54
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Sep 5 13:04:34 2023 -0400

    Merge pull request #23 from pclucas14/platypus-clean-monitors

    Platypus clean monitors

commit a89718b91914da2fdbe63cdbe4e56b1683842835
Author: oleksost <[email protected]>
Date:   Tue Sep 5 16:52:03 2023 +0000

    per layer selector logging

commit 2906f5fa34984b2da0f812c732efc0d08385054d
Author: oleksost <[email protected]>
Date:   Tue Sep 5 16:48:11 2023 +0000

    merging platypus

commit 688a3d317fa55da9c135c1405ccdf09c5429265d
Merge: 848100b ce0fade
Author: oleksost <[email protected]>
Date:   Tue Sep 5 16:47:44 2023 +0000

    Merge branch 'platypus' into platypus_vsmear_w_rgulaizer

commit ce0fade3a6386054afc5f9bc1164c09997a0f639
Merge: 28f1c91 ec74a54
Author: oleksost <[email protected]>
Date:   Tue Sep 5 16:44:55 2023 +0000

    Merge branch 'platypus-clean-monitors-local' into platypus

commit 28f1c919b9568b83f1103863a6291b4718275bbf
Author: oleksost <[email protected]>
Date:   Mon Sep 4 22:45:51 2023 +0000

    softmoe

commit 335fd2f4eea317a5b0aaf43fc4c6ac6a8e782393
Author: oleksost <[email protected]>
Date:   Mon Sep 4 22:45:38 2023 +0000

    softmoe

commit ec74a547852c31d5c3ca87fc4120103a44dd4e53
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Sep 4 22:07:23 2023 +0000

    subsample fix

commit bb93e9b42341bd8b35c82268997f4b4496edb4cf
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Sep 4 22:03:18 2023 +0000

    subsample n proportion per task

commit 7eac27986375bf79c6c636b40281e291ebc690af
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Sep 4 19:31:09 2023 +0000

    support for train_on_inputs

commit bff1cf39ffe1c217350c70774d84af758e639916
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Sep 4 19:30:55 2023 +0000

    sigh

commit d460edce95ec483648cc72ea4d952a024bac85ad
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Sep 4 16:58:06 2023 +0000

    fix

commit 8087975e6590d5cac5d4cc0a898263c1c3c9a377
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Sep 4 03:44:04 2023 +0000

    switch ni to original data module

commit 16da6e5f3ca5a994c6aa5f50e1b2b0e1e0096f7a
Author: Alessandro Sordoni <[email protected]>
Date:   Sun Sep 3 21:31:34 2023 +0000

    ni orig data mod supports finetune_task_name

commit e440999fc5c8129adb8386b673fd1ede21f5ace2
Author: Alessandro Sordoni <[email protected]>
Date:   Sun Sep 3 21:31:15 2023 +0000

    move decode in fn

commit 06a7e63a9d19158ce9b4bf455eb21a19b25810d0
Author: Alessandro Sordoni <[email protected]>
Date:   Sun Sep 3 04:14:24 2023 +0000

    handle few shot properly in ni

commit 0f471e7a1821e69d7030934b280c6a8b9acf459b
Author: Alessandro Sordoni <[email protected]>
Date:   Sun Sep 3 04:13:42 2023 +0000

    cut max instances per task only on train/valid for sni, not on test

commit 35cb8b4d3b4c3d5f4048d5247fad76c5f0bd6d99
Author: Alessandro Sordoni <[email protected]>
Date:   Sun Sep 3 03:37:26 2023 +0000

    subsample

commit 374795547e132a129fc0a0754f108d0f68d7b5ca
Author: Alessandro Sordoni <[email protected]>
Date:   Sun Sep 3 03:36:39 2023 +0000

    rename to task_to_id

commit 439a7cf5b74cc92b2c4a2f4d0e716ada88748a02
Author: Alessandro Sordoni <[email protected]>
Date:   Sun Sep 3 03:36:27 2023 +0000

    support for finetune_task_name ...

commit 5ad7ffa94f3e4c1f08576ae89a786d57fa6ebc94
Author: Alessandro Sordoni <[email protected]>
Date:   Sun Sep 3 03:36:15 2023 +0000

    setup_dataset

commit 046b49725193436328d80b1acdce5ab958af81af
Author: Alessandro Sordoni <[email protected]>
Date:   Sun Sep 3 03:34:01 2023 +0000

    fix bug in label_smoothed_nll_loss when ignore_index is negative

commit 223b42c754a8362965df1f304ec3f7a6aba168a9
Author: Alessandro Sordoni <[email protected]>
Date:   Sun Sep 3 03:17:06 2023 +0000

    setup logging for mhr

commit 80a4fadd65430298d113505263d479520fa6bc76
Author: Alessandro Sordoni <[email protected]>
Date:   Sun Sep 3 03:05:21 2023 +0000

    cleanup

commit 835751dec0d9da04a619d9787163c09c32aa51f8
Author: Alessandro Sordoni <[email protected]>
Date:   Sun Sep 3 03:03:47 2023 +0000

    train_dir -> data_dir

commit 8b90ef558690b410eff187ae3e5c877d7ccf539d
Author: Alessandro Sordoni <[email protected]>
Date:   Sun Sep 3 03:02:55 2023 +0000

    rename method

commit 0e172e89fa655aa32a417cec2944cd076dbdc086
Author: Alessandro Sordoni <[email protected]>
Date:   Sun Sep 3 02:53:34 2023 +0000

    lil bit of cleaning

commit 02d68eaf6e4430dae72f27d7bf79eaee66a0eeaa
Author: Alessandro Sordoni <[email protected]>
Date:   Sun Sep 3 02:37:40 2023 +0000

    save stddev

commit 722ff55d41d3fcbbc6af476fe1adffd945e4328f
Author: Alessandro Sordoni <[email protected]>
Date:   Sat Sep 2 23:13:38 2023 +0000

    monitor modif

commit 40a194705b3e13f6031ca2b70d5ef84d76f453fd
Author: Alessandro Sordoni <[email protected]>
Date:   Sat Sep 2 23:11:42 2023 +0000

    finish refact

commit 15e929e5917e0611b9c8c98b9bc830b8f1a3ff23
Author: Alessandro Sordoni <[email protected]>
Date:   Sat Sep 2 21:49:37 2023 +0000

    delimiter

commit f702a14bcd3c9124b3ba7a319ddb425d435bf1f9
Author: Alessandro Sordoni <[email protected]>
Date:   Sat Sep 2 21:40:37 2023 +0000

    fix var name

commit 0668100e81f28834fa563328b31b2eecffcf107a
Author: Alessandro Sordoni <[email protected]>
Date:   Sat Sep 2 21:40:12 2023 +0000

    adjust for mean

commit 8abc1851c08c0e435c79af9515c25277b41fe6b9
Author: Alessandro Sordoni <[email protected]>
Date:   Sat Sep 2 21:36:03 2023 +0000

    setup logging accepts log_dir = None

commit 9528b39d0b965e60a9d2ecbd1ac769038b0018c7
Author: Alessandro Sordoni <[email protected]>
Date:   Sat Sep 2 21:34:44 2023 +0000

    use mttl logger everywhere

commit 565638386725b6c7b5871b92a255028e600bf619
Author: Alessandro Sordoni <[email protected]>
Date:   Sat Sep 2 21:32:03 2023 +0000

    task aggregation

commit 8ebcff37c927ddb15521191b000c9f2d5551b917
Author: Alessandro Sordoni <[email protected]>
Date:   Sat Sep 2 21:28:00 2023 +0000

    mean_stderr

commit 41ad3805b96d8ddc3286451bc39d83af5e676ad0
Author: Alessandro Sordoni <[email protected]>
Date:   Sat Sep 2 21:09:59 2023 +0000

    dumps->dump

commit 50efdb0729ee1a523950084cc6778173bdc8c4f5
Author: Alessandro Sordoni <[email protected]>
Date:   Sat Sep 2 21:02:58 2023 +0000

    remove min/max on monitor

commit 7f44234bd8a1e16135f4b7aee8a1c850b6a9abcc
Author: Alessandro Sordoni <[email protected]>
Date:   Sat Sep 2 21:00:42 2023 +0000

    move model to gpu?

commit 4278807dbf71e9c816c83c14b0e1dac3b5ea80f9
Author: Alessandro Sordoni <[email protected]>
Date:   Sat Sep 2 20:55:09 2023 +0000

    dist utils only eval on main process

commit d48458cf0d70eb6c5ecf026a94e3862e5cc32ae4
Author: Alessandro Sordoni <[email protected]>
Date:   Sat Sep 2 20:40:57 2023 +0000

    save all metrics for mmlu/ni

commit 39437af0a2ce344c48c15b2f325fb250e18003c4
Author: Alessandro Sordoni <[email protected]>
Date:   Sat Sep 2 20:40:30 2023 +0000

    remove metric per task

commit c53d815db668cf6fd7038ea900734249d84644ae
Author: Alessandro Sordoni <[email protected]>
Date:   Sat Sep 2 20:23:48 2023 +0000

    turn off evals for now

commit 41f87f5dd195ddfc60c435c20c27a50ce9596bac
Author: Alessandro Sordoni <[email protected]>
Date:   Sat Sep 2 20:21:03 2023 +0000

    just indents

commit 883ebb1f1e4a897c863ab55275e8bb3acad89db4
Author: Alessandro Sordoni <[email protected]>
Date:   Sat Sep 2 20:20:41 2023 +0000

    mean over aux losses

commit 80f0ae6ff5489c02779de1b650ac0ce2d7b7298e
Author: Alessandro Sordoni <[email protected]>
Date:   Sat Sep 2 00:35:36 2023 +0000

    fix ni decoding when not using gpt

commit a21142824a511544b86972dc90cb985dc751f425
Author: Alessandro Sordoni <[email protected]>
Date:   Fri Sep 1 20:31:10 2023 +0000

    cache inputs for generation into routing infos

commit 82874cd3e0b33d5d5b49a5ef9b1bb29b7273bf68
Author: Alessandro Sordoni <[email protected]>
Date:   Fri Sep 1 19:41:52 2023 +0000

    clear cache

commit 7653b4c7bd8386d88ef73fa3748fe71a6c8e1b8a
Author: Alessandro Sordoni <[email protected]>
Date:   Fri Sep 1 19:13:08 2023 +0000

    vsmear fix norm

commit f303f554b8c988c30918a15b63d9e82491926133
Author: Alessandro Sordoni <[email protected]>
Date:   Fri Sep 1 19:13:02 2023 +0000

    centering for smear helps routing mi

commit 7a0de493785d66936b4cad4e192a99cb77bb7446
Author: Alessandro Sordoni <[email protected]>
Date:   Fri Sep 1 17:13:38 2023 +0000

    some loggin' for smear

commit 86c1cf57ae4b9dd3b492f38749ba46ebeb8cd5b2
Author: Alessandro Sordoni <[email protected]>
Date:   Fri Sep 1 17:06:21 2023 +0000

    dino-v2 type centering

commit 338b554302fb5c0af40f76f18c9617b1041cbedd
Author: Alessandro Sordoni <[email protected]>
Date:   Fri Sep 1 16:35:12 2023 +0000

    smear kl term

commit 17d60b77dbf74c26815d82c3212b7390eeb6b84c
Author: Alessandro Sordoni <[email protected]>
Date:   Fri Sep 1 16:33:57 2023 +0000

    unused imports

commit b7c4e2e7d51e6508cfa2805c864feb5b89e0f51e
Author: Alessandro Sordoni <[email protected]>
Date:   Fri Sep 1 16:31:50 2023 +0000

    l2 weight normalization

commit ee4e250ff84456db87cc290f61cb826439354faf
Author: Alessandro Sordoni <[email protected]>
Date:   Fri Sep 1 16:21:18 2023 +0000

    averager don't start from 0

commit 70d45361d5dab553229d132e10932dbf70767d18
Author: Alessandro Sordoni <[email protected]>
Date:   Fri Sep 1 14:55:52 2023 +0000

    fix routing info bug

commit 848100b698796432b593ef4b0525e52376dc574f
Author: oleksost <[email protected]>
Date:   Fri Sep 1 14:34:53 2023 +0000

    wip

commit cec30f187f6d3cc1a48fbe0365625cc803f6f065
Author: Alessandro Sordoni <[email protected]>
Date:   Fri Sep 1 14:27:16 2023 +0000

    fix args.tensorboard missing

commit d6f3345454e5f851acebfa068731bd7c8b7fff06
Merge: a90d42e d7146cf
Author: oleksost <[email protected]>
Date:   Fri Sep 1 13:21:01 2023 +0000

    Merge branch 'platypus-clean-monitors' into platypus_vsmear_w_rgulaizer

commit a90d42eefc2ca76e1d94390e4212a3261f9c85a6
Author: oleksost <[email protected]>
Date:   Fri Sep 1 13:17:06 2023 +0000

    wip

commit d7146cf6c214190e2420078b9ee85a1e9ae56fb6
Author: Alessandro Sordoni <[email protected]>
Date:   Fri Sep 1 03:33:31 2023 +0000

    correct averager

commit c1d6d824c5773dc3f61df19b1e9a04d0e8d3c4e0
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Aug 31 19:36:01 2023 +0000

    fix eta

commit fb59c2cccd41ed1a746833c008fe7cc854a6364e
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Aug 31 19:02:01 2023 +0000

    remove

commit 2686f1ea1d7fe473a5d4b48218267fee89f5c237
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Aug 31 18:58:00 2023 +0000

    back-compatibility

commit d32e46265a0b3c23ed08b603fc6d49454932e747
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Aug 31 18:52:59 2023 +0000

    eta

commit 9f41924aeb3e4de5141342bf19af8abfa4c9348c
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Aug 31 16:12:21 2023 +0000

    need to clone torch vars

commit b5fdb00867237a4f8efd6a46c742c8871f614110
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Aug 31 15:56:51 2023 +0000

    metric sugar

commit a366fc755e9d88076ce2e9bb2f67b44b95e30232
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Aug 31 15:48:44 2023 +0000

    min, max, avg agg of dicts

commit c8f62aeddd0be538f2c3537e19e31d515c59e9ac
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Aug 31 15:34:18 2023 +0000

    rename fix

commit fc76c2861630ed9024f1a071eb0fc8c0e091501d
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Aug 31 15:33:52 2023 +0000

    simplify ema

commit b11f94c2cfa0f17dfb2f0daccf55fe181a9a3f5e
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Aug 31 15:30:05 2023 +0000

    fix ema running avg

commit d047e6005394e616d99f0c98f990776d326ee45a
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Aug 31 15:10:55 2023 +0000

    use mini progress bar

commit 3fbaff3fa297f035f63240afcd42f7f914fca6fc
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Aug 31 15:10:46 2023 +0000

    all diversity in monitors

commit b09cf836a437cc9ce2d9862e3159d12f6a813837
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Aug 31 15:10:29 2023 +0000

    fix logging

commit 0962270d839bed2c837839a17cc8836b12f554ef
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Aug 31 15:10:24 2023 +0000

    fix logging and nanoprogress

commit d5c573227fdcc31e78762c0164d340b9ff0096d4
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Aug 31 10:42:50 2023 +0000

    fix

commit 7d37e69d84173416bd0f464b79332a0e20d018cd
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Aug 31 10:40:07 2023 +0000

    fix

commit 87e12b7d3f21b87e3d5f2fe47a8120dd3ffef6c3
Author: Alessandro Sordoni <[email protected]>
Date:   Thu Aug 31 10:39:49 2023 +0000

    fix

commit 8860039304017be0e862b730faa90dc2a9c6dcf7
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 20:25:20 2023 -0700

    metrics in monitors

commit 2d1287f5e8283f7f373bda178d3a7b70e96382d1
Author: oleksost <[email protected]>
Date:   Thu Aug 31 02:59:29 2023 +0000

    wip

commit 9d1247a97d8ff6248369e94ec2b4e7089afb3a0b
Author: oleksost <[email protected]>
Date:   Thu Aug 31 02:57:51 2023 +0000

    wip

commit 99710bd8782e0d6c9cb24cf4ec37ef42999d91cb
Author: oleksost <[email protected]>
Date:   Wed Aug 30 23:41:25 2023 +0000

    vsmear_wreg

commit c48ba74f78de320adf994730cfaccb081473072f
Author: oleksost <[email protected]>
Date:   Wed Aug 30 23:02:20 2023 +0000

    mege platypus

commit d35ae82d4d7c1753e22f05581c9c5acb6378fdb0
Merge: b88270d cb58b8c
Author: oleksost <[email protected]>
Date:   Wed Aug 30 23:00:49 2023 +0000

    Merge branch 'platypus' into platypus_vsmear_w_rgulaizer

commit b88270d201bc4436a166513387a205eddd73ca51
Author: oleksost <[email protected]>
Date:   Wed Aug 30 22:59:27 2023 +0000

    wip

commit cb58b8c9de5f4e12d65c2fcc070d747b9ea37485
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 15:22:30 2023 -0700

    fix bug of double-backward, clear losses and metric in adapter, before forward

commit bcfb4a2a507c1fdd653805ab7d70dd9f095e1313
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 14:31:44 2023 -0700

    better tokenizer getter

commit 381d0f7a3cc0d95a4729292cbc7a1d9df3de675c
Merge: e941bc9 a8d3199
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 14:22:44 2023 -0700

    Merge branch 'platypus' of github.com:pclucas14/lucas_mttl into platypus

commit e941bc92b3779c78174a202e47fe41f03b26d6bc
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 14:22:42 2023 -0700

    healthy space

commit a8d319959925b33af214540d38568c34a0684ea4
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 21:13:14 2023 +0000

    output length for NI to 128

commit 383ced274199334f3b47ec0f813d2fbe286e4216
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 14:07:47 2023 -0700

    do not strip spaces, why are we even doing that

commit 458bb87649dfa21f43f688a53a32cfe709ef207f
Author: oleksost <[email protected]>
Date:   Wed Aug 30 19:57:27 2023 +0000

    wip

commit fe0a4a7eb08b309b8e705d567b993b12db82814d
Merge: 2dade84 9886958
Author: oleksost <[email protected]>
Date:   Wed Aug 30 19:34:50 2023 +0000

    Merge branch 'platypus' into platypus_vsmear_w_rgulaizer

commit 988695880672ceebd5156fc9c2f3f8e537565de9
Author: oleksost <[email protected]>
Date:   Wed Aug 30 19:34:25 2023 +0000

    keepdim

commit 2dade84ee0412c25e299d153bc3639d97cdbf8e0
Author: oleksost <[email protected]>
Date:   Wed Aug 30 19:33:59 2023 +0000

    wip

commit 91c4d288a4b98f588c48153c590ceca6210d832a
Merge: 17fd63c 6ae2981
Author: oleksost <[email protected]>
Date:   Wed Aug 30 19:10:15 2023 +0000

    Merge branch 'platypus' into platypus_vsmear_w_rgulaizer

commit 17fd63ca63171095458aa1e24114233e556b1679
Author: oleksost <[email protected]>
Date:   Wed Aug 30 19:10:09 2023 +0000

    wip

commit 6ae2981a896eb11b318c33c40ac575c293898356
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 19:09:13 2023 +0000

    200 batches for mmlu

commit 74b80a447f5531014abc4710dc4be9e416a7b8b2
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 19:06:56 2023 +0000

    fix padding ne

commit 468ad1791a9e5e1513cc2132eb4d29ac2b60385f
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 18:58:26 2023 +0000

    fix hf token

commit 99927ab2a4159ec709dcf895169513e767cd8427
Merge: 0850f50 a72bb0b
Author: oleksost <[email protected]>
Date:   Wed Aug 30 18:44:14 2023 +0000

    Merge branch 'platypus' into platypus_vsmear_w_rgulaizer

commit a72bb0b47054cdafe115fb446747111d9b2a282f
Author: oleksost <[email protected]>
Date:   Wed Aug 30 18:41:44 2023 +0000

    cashing sequence for gen mode

commit 43c91c594b26630864e0df121f02085a17d408c6
Merge: 93222e7 1cf7f4e
Author: oleksost <[email protected]>
Date:   Wed Aug 30 18:40:45 2023 +0000

    Merge branch 'platypus' of https://github.com/pclucas14/lucas_mttl into platypus

commit 1cf7f4e3c9f8c5bffb5655c7200fda37d35c9d95
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 18:39:35 2023 +0000

    ok, ni_dataset got right

commit 93222e7296450cad0103a5dc204b7a7d9b311b43
Merge: 94d318d a3cfe80
Author: oleksost <[email protected]>
Date:   Wed Aug 30 18:34:09 2023 +0000

    Merge branch 'platypus' of https://github.com/pclucas14/lucas_mttl into platypus

commit a3cfe806794c367e54ae6f7026363e5ac71e9a3a
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 11:33:04 2023 -0700

    fix ni dataset

commit 94d318df19d80a6e9fb9d7e0c90cddc64164e2b4
Author: oleksost <[email protected]>
Date:   Wed Aug 30 18:20:19 2023 +0000

    cash sequence at gen time

commit 27469f02dbad1130e32739af49ca145a3c3453ba
Merge: d0c611b 339e3db
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 11:19:43 2023 -0700

    Merge branch 'platypus' of github.com:pclucas14/lucas_mttl into platypus

commit d0c611b4abe1519f626a846b3ae8689b8c57ee92
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 11:19:37 2023 -0700

    ni callback

commit 61929e59dd9801bb56d70f135e6d0ae2ebf5996a
Author: oleksost <[email protected]>
Date:   Wed Aug 30 18:13:12 2023 +0000

    cash sequence at generation time

commit 339e3dbe4e1e8485a7c938d9690f329ec60166fb
Merge: db2f6f5 3a3ebdf
Author: Oleksiy Ostapenko <[email protected]>
Date:   Wed Aug 30 13:59:30 2023 -0400

    Merge pull request #22 from pclucas14/platypus_flan

    added flan dataset and module

commit 3a3ebdf75f4dd5c1439cbfce655370480cd6de04
Merge: a83b9ad db2f6f5
Author: Oleksiy Ostapenko <[email protected]>
Date:   Wed Aug 30 13:59:16 2023 -0400

    Merge branch 'platypus' into platypus_flan

commit db2f6f51d881d59200e77592e3e0786f8541f1c7
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 10:42:04 2023 -0700

    eval_mmlu

commit 6c55133faf6104b577b3f5504516e483ac366c70
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 10:39:12 2023 -0700

    found mmlu bug, wasn't scoring the correct options

commit 0850f50c399cfd91cde649899d1ec66eaa1265a5
Merge: 46678b3 afab6d1
Author: oleksost <[email protected]>
Date:   Wed Aug 30 17:14:56 2023 +0000

    Merge branch 'platypus' into platypus_vsmear_w_rgulaizer

commit 46678b3a7400412ca387839d18c9a495d9800cda
Author: oleksost <[email protected]>
Date:   Wed Aug 30 17:14:03 2023 +0000

    wip

commit afab6d13b43d704e38356fbb0adbfc0080b5cbc7
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 15:39:41 2023 +0000

    predict batch size 2

commit d4d924c59f75e17c704001c33c89a5ff805a5cf2
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 15:32:37 2023 +0000

    set model.eval() in evaluators

commit e959caefb7c212485e6f60c762356c931946c41d
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 15:27:08 2023 +0000

    tensorboard flag, better handling in evaluators

commit a079c3f51ff2aa6ef346384014c1785f4b65b0a5
Author: oleksost <[email protected]>
Date:   Wed Aug 30 14:10:13 2023 +0000

    ni_dataset path bug

commit 9f41ba1f923f41f935222cab2f1485b9df5d3fe8
Author: oleksost <[email protected]>
Date:   Wed Aug 30 14:06:02 2023 +0000

    wip

commit a83b9ad8966764090c552b3ec9d36504096d59c4
Author: oleksost <[email protected]>
Date:   Wed Aug 30 12:20:58 2023 +0000

    added flan dataset and module

commit 0b8e22d6d644b9848a3a64785313e9d58d6b1ab1
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 02:24:54 2023 +0000

    llama2 13b

commit c631a971afefd3419b0c4ccc6f1138582b439480
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 02:24:43 2023 +0000

    8 skills by default

commit efa1a976c1a62ba770543b57330df79a7d0c751b
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 02:14:59 2023 +0000

    nit

commit 34009b761102cb589dc32314b8077d768e9480cc
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 02:13:46 2023 +0000

    nit softmoe

commit 928f8f97f8504b915888804cf3d5dbc0afaf36c7
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 02:12:11 2023 +0000

    indent w black

commit 16372c880c54521201fa3060732eafbcbf1e668a
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 02:01:58 2023 +0000

    nits

commit 28132c2518bda15f56a7580fc674e197fa882c05
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 01:59:59 2023 +0000

    setup eval callbacks

commit 6caaa64973895a227bf6aa846bdb683610108657
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 01:59:12 2023 +0000

    handy callbacks

commit 81f7382dcae188ec0b63c531175000c60ee7b3d1
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 01:52:06 2023 +0000

    black

commit 9586c7a26a4670f2da3e350ac8edef6cbdc00858
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 01:51:48 2023 +0000

    fix while in mmlu

commit 078935e747c323b250cefaba8dca26518cc56d4d
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 01:48:31 2023 +0000

    use else in get tokenizer

commit 05cad15f64efee07f40345f862f71d3907a7ed95
Author: Alessandro Sordoni <[email protected]>
Date:   Wed Aug 30 01:48:21 2023 +0000

    mmlu collator bugged when too long

commit 01cba0bf0506032697c1ff0f227ed0cf4b1d8e28
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 21:47:37 2023 +0000

    logger

commit 304a37fb34655c5b9289a3d46fdfb3f2dbe63979
Merge: 53ff7c9 e935740
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 17:42:39 2023 -0400

    Merge pull request #21 from pclucas14/platypus_softmoe

    Platypus softmoe

commit e9357401d1d27a225f48d2f3822eaabce1a693c5
Merge: f18652e 53ff7c9
Author: Oleksiy Ostapenko <[email protected]>
Date:   Tue Aug 29 17:29:12 2023 -0400

    Merge branch 'platypus' into platypus_softmoe

commit f18652e66c82b644bf27ea1c31d5cb1d7728655d
Author: oleksost <[email protected]>
Date:   Tue Aug 29 20:58:09 2023 +0000

    added softmoe like routing

commit 53ff7c9fe2c12404989bb338374f2753b71e2bf4
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 20:18:38 2023 +0000

    smooth length

commit 2149aed7ff89ab5c7e406e8c1e887587049c5d5d
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 20:15:05 2023 +0000

    input length

commit 76790fd66329402b7b3f610782bb8eb811743649
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 20:14:42 2023 +0000

    fix logging

commit 2a8d5bf71f124950efa171c8420669630fc959f2
Merge: fe1ea65 4a72d23
Author: oleksost <[email protected]>
Date:   Tue Aug 29 17:55:21 2023 +0000

    Merge branch 'platypus' into platypus_softmoe

commit 4a72d23c5e390afa4782a73ffbf7802c66c18f07
Merge: afbb230 a787e2d
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 13:49:29 2023 -0400

    Merge pull request #19 from pclucas14/platypus_gpt

    Platypus gpt

commit a787e2de71782810c670d88f62b6605a9b09d081
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 17:42:22 2023 +0000

    multiply inst mask by padding!

commit 0dd3ca43751f74f3ec1c909c8890f935abd63752
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 17:38:06 2023 +0000

    implement smear

commit 82bfc1bb362f86c2840c89d085a2a5389324d599
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 17:27:07 2023 +0000

    ni generation

commit 662ef3f04d65f6323b829ae120ce3a00588baf17
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 17:25:38 2023 +0000

    shuffle for ni

commit f7b814a17831d48987e2711ac6a371cb527e43d4
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 17:24:39 2023 +0000

    gpt collator for mmlu

commit bf04d0c4da492463571e85d133826dcecfdbd05b
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 15:51:51 2023 +0000

    fix collators

commit 0b7edbdba535ae4424d9528c7828503ecb385220
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 14:15:40 2023 +0000

    dont detach

commit e9d9932f4e531077b454d7a0b9092e31e86534df
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 13:58:37 2023 +0000

    set pad_token_id to 0

commit 0a89395fe8a56bc73dd5b5222aee42f6c187eb66
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 13:42:28 2023 +0000

    remove imports

commit b1b3f0510f3d52b13b39f0ba27130e3f43ff173b
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 13:40:43 2023 +0000

    black formatting

commit b55d751e959158d30c01a77f0aa667ebc9ccf627
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 13:40:32 2023 +0000

    force eos token to fals

commit 0d2c21c822253ad3c3924685dea51aa13f8f4f38
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 13:40:22 2023 +0000

    add eos token to labels before tokenization!

commit 0ede5326281eda5dbebc10d65caa0ae6c63f1854
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 13:40:04 2023 +0000

    loss_for_keywords contd

commit 539f1e5ce1517e4b55f43dc2766fea84476b3b8a
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 13:39:48 2023 +0000

    clean loss_for_keywords

commit b26b61e0ec02c88e6bc5cb9da4c738169f88f399
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 13:39:39 2023 +0000

    remove text only

commit 7690ddd2858d3b4edbe3002e9392858519965fc4
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 12:43:54 2023 +0000

    remove breakpoint

commit 6a17b94e7802e140829d4b5a772df42ad7ab02b5
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 12:43:26 2023 +0000

    prepare for gpt in default collator

commit 10ee0e757180e32101345d30a23b4d04c6219aa5
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 12:36:01 2023 +0000

    fixin

commit 367333f7df1a5856c8bd7363dc58ed6d09f297d5
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 12:35:22 2023 +0000

    fix cont'd

commit fe1ea65c0cd70b43391a27dde5f1bb3735108721
Merge: 86afe40 afbb230
Author: oleksost <[email protected]>
Date:   Tue Aug 29 12:27:08 2023 +0000

    merge plotypus

commit 86afe40e1ed6ed36c80fbf65681eb13de5161516
Author: oleksost <[email protected]>
Date:   Tue Aug 29 12:23:42 2023 +0000

    wip

commit 15d303b95e48053548a49da95c9498783bd93d7f
Author: Alessandro Sordoni <[email protected]>
Date:   Tue Aug 29 04:50:41 2023 +0000

    fix gpt family collator

commit afbb23079e565b134bb185a1fcc336f72612357b
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 23:00:35 2023 +0000

    adapter

commit 033ebfeb3e21810413b3f49c9e92dcfbdf936595
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 22:59:33 2023 +0000

    missing self

commit 28f3ff4dae2f9aefb5cbd01c4e6a84701bce647d
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 22:54:36 2023 +0000

    fix vsmear to track changes in mhr branch

commit 874492701c03e277d763c5d62813853f7102603b
Author: oleksost <[email protected]>
Date:   Mon Aug 28 22:51:41 2023 +0000

    wip

commit 469202b4076da3008d38f816fdf2dba9fb4ceaac
Merge: 7445ef7 cb9a544
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 18:49:52 2023 -0400

    Merge pull request #18 from pclucas14/platypus_mhr

    Platypus mhr

commit cb9a544d2e194191475c1e4880b44e8052fbe84c
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 22:49:41 2023 +0000

    lagging rename

commit 8d52dfe12dbbcc5d01be4827fc10d04b6ef5b6c6
Merge: 12e3518 7445ef7
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 18:48:29 2023 -0400

    Merge branch 'platypus' into platypus_mhr

commit 12e351843cde0eb48da1c579876df164b01fc043
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 22:46:38 2023 +0000

    fix self.weight error for t5

commit 7445ef7ccf06970e5d5e92f8dbb55d33b625ceee
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 22:42:44 2023 +0000

    nitty fix monitor

commit 8ecca9529858f4f363093f21fc169f48cf4d3a10
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 21:49:37 2023 +0000

    task_id to 0

commit 11b7f0f59588afe2b715384057dcf1c16ee2fe22
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 21:45:27 2023 +0000

    alpaca cleanup

commit ea29c998dea429aa07ced5ab47db0f39558761d8
Author: Lucas Caccia <[email protected]>
Date:   Mon Aug 28 21:44:22 2023 +0000

    default value for `router_selector` in poly_lora.json

commit 4190131528f4ecfb449cf02f82e53e6516fac0fa
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 21:43:39 2023 +0000

    nit fix alpaca data module

commit f57e93e75eb560e873d22e3f2474d35150934658
Author: Lucas Caccia <[email protected]>
Date:   Mon Aug 28 21:42:54 2023 +0000

    patch PolyLoRALinear to have a `weight` attribute

commit 889e46a841f09b8194b3ba3d9663e2116a5444ad
Author: Lucas Caccia <[email protected]>
Date:   Mon Aug 28 21:42:29 2023 +0000

    was missing `lora_dropout`

commit 959faf0fd0485e1d7e739c4926fcd48d5e686ada
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 17:15:03 2023 +0000

    fix sys path append

commit 988d3ac6f167ebe6b4827f9d9a2334772f84edcf
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 17:09:10 2023 +0000

    lora alpha 16

commit 20f1287b102012d27c74bc336ade8b9b88e8d0d0
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 16:59:57 2023 +0000

    nit config

commit 78d220232aa62f5ab57b5607e923aec20769b156
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 16:56:47 2023 +0000

    temperature for router

commit 743923092e7a65da97d4c32713a955b76a4eecd2
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 16:56:35 2023 +0000

    use logging

commit 7c188529128b7175d9c3dbb14efdf998d6c3e2c7
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 16:55:33 2023 +0000

    router temperature

commit 4389ded612fe975957d397d86323b182a37b9f20
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 16:51:22 2023 +0000

    don't detach just yet

commit 6c890c6e1147219fe61b7a69c7ccc43746dacce7
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 15:59:16 2023 +0000

    add router to params!!!

commit fe664dae10f89def2c12b8daf90bac004da462cc
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 15:55:25 2023 +0000

    fix formatting

commit fac5502869a04376b1c76bf02311d00eac1fcf86
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 15:53:58 2023 +0000

    setup_logging in mttl.utils

commit e4ef3bf62bcfb973b8183813fbac4430b9b16d3f
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 15:46:32 2023 +0000

    remove use_test_set

commit e0d3a5df44d06b2acd245ea038a8577c97e4e8b9
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 15:45:49 2023 +0000

    additional cleanup

commit a4c7b964bd385e6fd3f3fad4939b6e36180bd360
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 13:10:29 2023 +0000

    remove temp

commit 0e7017dd0df5480a6ebf8c348532e6d383cc4068
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 13:07:07 2023 +0000

    vsmear

commit d9be8c020a8059527be9492c856325782ef94238
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 13:04:21 2023 +0000

    data checks

commit 26267c74ff8b06b2211953e7702dd8381d770a18
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 13:02:28 2023 +0000

    import

commit bc9a819b8af90b3e9685620048351ea72456ff18
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 13:02:08 2023 +0000

    unused files

commit 2b42fe30a20124ded177d9a7f929aa73113e3103
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 12:59:44 2023 +0000

    transfer to device

commit 1215b6e7a9a72ce835fb197322e4dbed5273c9e6
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 03:41:15 2023 +0000

    matmul

commit 9741dbf0db66d56268b311e821dac3b67c883890
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 03:38:31 2023 +0000

    rank <- lora_rank

commit 19fcfda490d58667d42aac945ca497fff851b7a5
Author: Alessandro Sordoni <[email protected]>
Date:   Mon Aug 28 03:36:27 2023 +0000

    other big refactor: use adapter klass

commit…
Loading branch information
matheper committed Feb 14, 2024
1 parent ce4ca51 commit c66b20d
.gitignore
-Original file line number
+Diff line change
@@ -1,4 +1,5 @@
     # Project related stuff
+    configs/
     dataloader/*_data/processed/
     amulet_*/
     wandb
@@ Expand Down Expand Up / @@ -140,6 +141,16 @@ venv.bak/ @@
     .mypy_cache/
     .dmypy.json
     dmypy.json
     # Pyre type checker
-    .pyre/
+    .pyre/
+    **/eval/alpaca_eval/*
+    **/eval/alpaca_eval/**/*.csv
+    **/eval/alpaca_eval/**/*.json
+    **/eval/mmlu/**/*.csv
+    results/
+    lm_cache/
+    /tmp/
+    # any file starting with _
+    _*.*
mttl/callbacks.py
-Original file line number
+Diff line change
@@ -1,9 +1,163 @@
-    import sys
+    import datetime
+    import time
+    import sys, os
+    from typing import Any
     from pytorch_lightning import callbacks as cb
     from pytorch_lightning.callbacks.progress.tqdm_progress import Tqdm
+    from mttl.utils import Averager, logger
+    class MMLUCallback(cb.Callback):
+        def __init__(self, every_val_epochs=1, **kwargs):
+            super().__init__()
+            self.val_epoch = 0
+            self.every_val_epochs = every_val_epochs
+            self.eval_kwargs = kwargs
+        def on_validation_epoch_end(self, trainer, pl_module) -> None:
+            self.val_epoch += 1
+            if trainer.global_step == 0:
+                return
+            if self.val_epoch % self.every_val_epochs != 0:
+                return
+            from mttl.evaluators import MMLUEvaluator
+            evaluator = MMLUEvaluator(
+                pl_module.hparams,
+                data_dir=os.environ["MMLU_DATA_DIR"],
+                **self.eval_kwargs,
+            )
+            metrics = evaluator.evaluate(
+                pl_module,
+                subsample=10,
+            )
+            pl_module.log(
+                "val/mmlu",
+                metrics["all"]["mean"],
+                on_step=False,
+                on_epoch=True,
+                prog_bar=True,
+            )
+    class NICallback(cb.Callback):
+        def __init__(self, every_val_epochs=1, **kwargs):
+            super().__init__()
+            self.val_epoch = 0
+            self.every_val_epochs = every_val_epochs
+            self.eval_kwargs = kwargs
+        def on_validation_epoch_end(self, trainer, pl_module) -> None:
+            self.val_epoch += 1
+            if trainer.global_step == 0:
+                return
+            if self.val_epoch % self.every_val_epochs != 0:
+                return
+            from mttl.evaluators import NIEvaluator
+            evaluator = NIEvaluator(
+                pl_module.hparams,
+                data_dir=os.environ["NI_DATA_DIR"],
+                num_pos_examples=2,
+                **self.eval_kwargs,
+            )
+            metrics = evaluator.evaluate(
+                pl_module,
+                eval_batches=50,
+            )
+            pl_module.log(
+                "val/sni",
+                metrics["all"]["mean"],
+                on_step=False,
+                on_epoch=True,
+                prog_bar=True,
+            )
+    class MiniProgress(cb.ProgressBar):
+        def __init__(self):
+            super().__init__()
+            self.averager = Averager(0.9)
+        def on_train_batch_start(
+            self, trainer, pl_module, batch: Any, batch_idx: int
+        ) -> None:
+            self.time_start = time.time()
+        def on_train_batch_end(
+            self,
+            trainer,
+            pl_module,
+            outputs,
+            batch,
+            batch_idx,
+        ) -> None:
+            self.time_end = time.time()
+            metrics = self.get_metrics(trainer, pl_module)
+            metrics = {
+                k.replace("_step", "").replace("_epoch", ""): v for k, v in metrics.items()
+            }
+            it_per_sec = 1 / (self.time_end - self.time_start)
+            eta = (trainer.num_training_batches - batch_idx) / (
+.0 / ((self.time_end - self.time_start))
+            )
+            time_metrics = self.averager.update({"it/s": it_per_sec, "eta": eta})
+            for k, v in {**metrics, **time_metrics}.items():
+                if k == "eta":
+                    metrics[k] = "{}".format(datetime.timedelta(seconds=v))
+                else:
+                    metrics[k] = "{:.2f}".format(v) if isinstance(v, float) else v
+            msg_start = (
+                f"Trn - Epc {trainer.current_epoch} / {trainer.global_step} / {trainer.num_training_batches // trainer.accumulate_grad_batches}"
+                + " | "
+            )
+            dict_msg = " | ".join([f"{k} -> {v}" for k, v in metrics.items()]) + " | "
+            msg = msg_start + dict_msg
+            logger.info(msg)
+        def on_validation_batch_start(
+            self, trainer, pl_module, batch: Any, batch_idx: int
+        ) -> None:
+            self.time_start = time.time()
+        def on_validation_batch_end(
+            self,
+            trainer,
+            pl_module,
+            outputs,
+            batch,
+            batch_idx,
+        ) -> None:
+            self.time_end = time.time()
+            metrics = self.get_metrics(trainer, pl_module)
+            metrics = {
+                k.replace("_step", "").replace("_epoch", ""): v for k, v in metrics.items()
+            }
+            metrics["it/s"] = 1.0 / (self.time_end - self.time_start)
+            for k, v in metrics.items():
+                metrics[k] = "{:.2f}".format(v) if isinstance(v, float) else v
+            msg_start = (
+                f"Val - Epc {trainer.current_epoch} / {batch_idx} / {trainer.num_val_batches[0]}"
+                + " | "
+            )
+            dict_msg = " | ".join([f"{k} -> {v}" for k, v in metrics.items()]) + " | "
+            msg = msg_start + dict_msg
+            logger.info(msg)
     class ProgressCallback(cb.TQDMProgressBar):
         def init_sanity_tqdm(self) -> Tqdm:
             """Override this to customize the tqdm bar for the validation sanity run."""
@@ Expand Down @@
0 comments on commit `c66b20d`

Please sign in to comment.
Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Commit

There are no files selected for viewing

0 comments on commit `c66b20d`

Commit

There are no files selected for viewing

0 comments on commit c66b20d

0 comments on commit `c66b20d`
