1(window.webpackJsonp=window.webpackJsonp||[]).push([[0],{142:function(e){e.exports={apiKey:"AIzaSyBEFTw0-JSm_c5liHBCbeLubgAZhSDj8lU",authDomain:"mtl-sentence-representations.firebaseapp.com",databaseURL:"https://mtl-sentence-representations.firebaseio.com",projectId:"mtl-sentence-representations",storageBucket:"mtl-sentence-representations.appspot.com",messagingSenderId:"705560051964",basename:"/"}},203:function(e,t){e.exports="#### Table of Contents\n- [Introduction](#introduction)\n- [Contents](#contents)\n- [Usage](#usage)\n- [Format](#format)\n- [Design](#design)\n - [Standards for entailment](#standardsforentailment)\n - [Handling Coreference](#handlingcoreference)\n - [Definite Descriptions and Monotonicity](#definitedescriptionsandmonotonicity)\n - [Background Knowledge](#backgroundknowledge)\n- [Linguistic Categorization](#linguisticcategorization)\n - [Lexical Semantics](#lexicalsemantics)\n - [Lexical Entailment](#lexicalentailment)\n - [Morphological Negation](#morphologicalnegation)\n - [Factivity](#factivity)\n - [Symmetry/Collectivity](#symmetrycollectivity)\n - [Redundancy](#redundancy)\n - [Named Entities](#namedentities)\n - [Quantifiers](#quantifiers)\n - [Predicate-Argument Structure](#predicateargumentstructure)\n - [Syntactic Ambiguity](#syntacticambiguityrelativeclausescoordination-scope)\n - [Prepositional Phrases](#prepositionalphrases)\n - [Core Arguments](#corearguments)\n - [Alternations](#alternationsactivepassivegenitivespartitivesnominalizationdatives)\n - [Ellipsis/Implicits](#ellipsisimplicits)\n - [Anaphora/Coreference](#anaphoracoreference)\n - [Intersectivity](#intersectivity)\n - [Restrictivity](#restrictivty)\n - [Logic](#logic)\n - [Propositional Structure](#propositionalstructurenegationdoublenegationconjunctiondisjunctionconditionals)\n - [Quantification](#quantificationuniversalexistential)\n - [Monotonicity](#monotonicityupwardmonotonedownwardmonotonenonmonotone)\n - [Richer Logical Structure](#richerlogicalstructureintervalsnumberstemporal)\n - [Knowledge and Common sense](#knowledgeandcommonsense)\n - [World Knowledge](#worldknowledge)\n - [Common Sense](#commonsense)\n\n## Introduction\n\nThe GLUE benchmark comes with a manually-curated evaluation dataset for fine-grained analysis of\nsystem performance on a broad range of linguistic phenomena.\n\nThis dataset evaluates sentence understanding through Natural Language Inference (NLI)\nproblems. The NLI task is well-suited to our purposes because it can encompass a large set of\nskills involved in language understanding, from resolving syntactic ambiguity to high-level\nreasoning, while still supporting a straightforward evaluation.\n\nSee Section 4 in [the GLUE paper](https://www.nyu.edu/projects/bowman/glue.pdf) for a description of\nthe annotation process and some examples. If you have questions or concerns about the diagnostic\nset, first consult this document, then the [FAQ](/faq), and if your questions aren't answered, ask on\nthe [Google Group](mailto:[email protected]). Download the diagnostics data with labels and tags from this [link](https://www.dropbox.com/s/ju7d95ifb072q9f/diagnostic-full.tsv?dl=1).\n\n## Contents\n\nThe data consists of several hundred sentence pairs labeled with their entailment relations\n(entailment, contradiction, or neutral) in both directions and tagged with a set of linguistic\nphenomena involved in justifying the entailment labels. It was constructed manually by the authors\nof GLUE and draws text from several different sources, including news, academic and encyclopedic\ntext, and social media. The sentence pairs were crafted so that each sentence in a pair is very\nsimilar to the other, to make the problem harder for systems that rely on simple lexical cues\nand statistics.\n\nLinguistic phenomena are tagged with labels coarse- and fine-grained categories. The coarse-grained\ncategories are [Lexical Semantics](#lexicalsemantics),\n[Predicate-Argument Structure](#predicateargumentstructure), [Logic](#logic), and\n[Knowledge and Common Sense](#knowledgeandcommonsense). Each of these has several fine-grained\nsubcategories for specific linguistic phenomena of that kind. See the\n[Linguistic Categorization](#linguisticcategorization) section for\ndetails on the meanings of all of the categories and how they are annotated.\n\n## Usage\n\nThe diagnotic set was labeled most closely to the schema of\n[MultiNLI](http://www.nyu.edu/projects/bowman/multinli/).\nYou should run the MultiNLI predictor of your model on the diagnostic data when submitting results.\n\nBecause the diagnostic examples are hand-picked to address certain phenomena, we expect that they\nwill not be representative of the distribution of language as a whole, even in the targeted domains.\nHowever, NLI is a task with no natural input distribution. We deliberately select sentences that we\nhope will be able to provide insight into what models are doing, what phenomena they catch on to,\nand where are they limited. This means that the raw performance numbers on the analysis set should\nbe taken with a grain of salt. The set is provided not as a benchmark, but as an analysis tool to\npaint in broad strokes the kinds of phenomena a model may or may not capture, and to provide a set\nof examples that can serve for error analysis, qualitative model comparison, and development of\nadversarial examples that expose a model's weaknesses.\n\nBecause the distribution is somewhat arbitrary, it will not be helpful to compare the performance\nnumbers of the same model on different categories. Rather, you should compare the numbers that\ndifferent models score on the same category, or use the reported scores as a guide to dig into\nspecific examples to do error analysis and understand your model better.\n\n## Format\n\nThe data is distributed in `.tsv` format with the following columns:\n\n0. Lexical Semantics\n1. Predicate-Argument Structure\n2. Logic\n3. Knowledge & Common Sense\n4. Domain\n5. Premise\n6. Hypothesis\n7. Label\n\nFor each entry, columns 0 through 3 c
1ontain a semicolon-separated list of fine-grained linguistic\ncategories (see [below](#linguisticcategorization) for the full list), column 4 is `News`,\n`Reddit`, `Wikipedia`, `ACL`, or `Artificial`, columns 5 and 6 are the (untokenized) sentences, and\ncolumn 7 is the entailment relation (`entailment`, `contradiction`, or `neutral`).\n\nSince we labeled each sentence pair in both directions, each pair appears twice (consecutively) in\nthe data, with the premise and hypothesis switched and a possibly different entailment label.\n\n## Design\n\nIn general, we regard the NLI problem as one of judging what a typical human reader would conclude\nto be true upon reading the premise, absent the effects of pragmatics. Inevitably there will be many\ncases which are not purely, literally implied, but we want to build systems that will be able to\ndraw the same conclusions as humans. Especially in the case of commonsense reasoning, which often\nrelies on defeasible inference, this will be the case. We try to exclude particularly questionable\ncases from the data, and we do not use any sentences that are ungrammatical or semantically\nincoherent.\n\nIn general, we use the standards set in the\n[RTE Challenges](https://link.springer.com/chapter/10.1007/978-94-024-0881-2_42)\nand follow the guidelines of\n[MultiNLI](http://www.nyu.edu/projects/bowman/multinli/).\nGiven two sentences (a _premise_ and _hypothesis_), we label them with one of three entailment\nrelations:\n\n* Entailment: the hypothesis states something that is definitely correct about the situation or\nevent in the premise.\n* Neutral: the hypothesis states something that might be correct about the situation or\nevent in the premise.\n* Contradiction: the hypothesis states something that is definitely incorrect about the situation or\nevent in the premise.\n\nThese definitions are essentially the same as what was provided to the crowdsourced annotators of\nthe MultiNLI dataset.\nThey rely on an assumption that the two sentences describe the same situation.\nHowever, a \"situation\" may involve multiple participants and actions, and the granularity at which\nwe ask the described situations to be the same is somewhat subjective.\n\nThe remainder of this section describes the decisions we made when constructing the diagnostic\ndataset to decide these issues.\n\n#### Handling Coreference\n\nSince our data consists of sentences taken out of context, they will often use referring expressions\nand anaphora that have no clear antecedent. In these cases, we assume that, where possible,\nexpressions in the premise and hypothesis are co-referent. For example, if there are definite noun\nphrases in each labeled _the squirrel_ and _the dog_, then we assume they refer respectively to the\nsame salient squirrel and dog, without causing an unnecessary contradiction.\n\nHowever, it's possible to get into tricky territory fairly quickly: Does _the squirrel is brown_\ncontradict _the animal is white_? It depends on whether you want to force coreference between\n_the squirrel_ and _the animal_, as the described situation may conceivably involve multiple\ncreatures. This can get hairier with event coreference:\nclearly _John likes Gary_ contradicts _John dislikes Gary_, because they both describe a state that\nholds at the present time. But how about _John gave Gary a dollar_ and _Gary gave John a dollar_?\nThey are contradictory if you force coreference of the _gave_ event, but not if you allow that they\nhappened in rapid sequence. And if you allow them to happen in sequence, perhaps\n_John gave Gary a dollar reluctantly_ should not contradict\n_John gave Gary a dollar enthusiastically_. So where is the bright-line?\n\nIn order to build a diverse set of examples using natural text, we had to be a bit flexible. So\nto determine coreference, we judge based on the amount of _corroborating information_ to a\ncoreference judgment. For a noun, the same noun phrase description is enough corroborating\ninformation to induce coreference. But for a verb, its core arguments (e.g., _John_, _Gary_, and\n_a dollar_) must be co-referent in order to induce the coreference. The exception is if there is\nextra corroborating information (for example, a temporal modifier or quantifier like _only_) that\nuniquely establishes the event denoted by the given verb. (This will often happen in our examples\nthat deliberately permute the arguments of verbs.)\n\nThis is not a perfect solution, and
1you may not agree with all of the judgments. If you wish to\ndo your own evaluation without worrying about these coreference problems, there is an easy solution:\ncollapse _neutral_ and _contradiction_ into a single _non-entailment_ class. This works because the\ntension with coreference is between forcing an _inadmissible_ coreference (i.e., one that leads to a\n_contradiction_) and allowing reference to a _new_ entity (which almost always leads to a _neutral_).\n\n#### Definite Descriptions and Monotonicity\n\nOur sentence pairs were constructed mostly by modifying sub-phrases of a premise to produce a\nhypothesis. Any time you wish to determine the entailment relation between two sentences that differ\nonly on a sub-phrase, the logical phenomenon of monotonicity will come into play (see the sections\non [Lexical Entailment](#lexicalentailment) and [Monotonicity](#monotonicity)). However, if we gave\nevery single case a monotonicity label, then the [Logic](#logic) category for coarse-grained\nanalysis would lose its usefulness. So instead, we only identify the broad categories of\n_upward monotone_, _downward monotone_, and _non-monotone_, and leave anything that does not\nobviously fall into these categories unlabeled.\nThe main example of this is the (fairly common) case of definite descriptor that picks out a\nparticular entity or set of entities to apply a property to.\nUsing the position of the word _mammals_ as an example:\n\n* Upward monotone: `Some squirrels are mammals.`\n* Downward monotone: `No squirrels are mammals.`\n* Non-monotone: `Exactly two squirrels are mammals.`\n* No monotonicity label (definite descriptor): `These squirrels are mammals.`\n\nSo not only does this approach help us avoid the blow-up of monotonicity labels, but in these\nparticular cases, in some sense we don't need them. The various kinds of monotonicity\nrelations block or shift entailments propagating up from their sub-phrases, but in the case of a\ndefinite description, the entailment relation passes up completely unchanged:\n`These squirrels are mammals`\nentails `These squirrels are animals`,\ncontradicts `These squirrels are fish`, and\nis entailed by `These squirrels are rodents`.\nYou can verify that these don't all three hold for any of the other quantifiers.\n\n#### Background Knowledge\n\nThe incorporation of background knowledge also presents challenges to formulating NLI. On one hand,\nwe want to capture that natural language understanding requires the incorporation of background\nknowledge. On the other hand, allow too much background knowledge and the entailment problem can\nhit degenerate cases. For example, if you know that Theresa May is the British Prime Minister, than\ntechnically any premise entails `Theresa May is the British Prime Minister`.\n\nHowever, since we are manually curating our data and our sentence pairs always refer to the same or\nsimilar situation, these cases do not end up being a problem. We can assume an\nabsolutely minimal amount of background knowledge, and in cases where some kind of knowledge\nis directly required to solve the entailment (notwithstanding lexical and syntactic disambiguation),\nwe can directly tag the example \u2014 usually with [Common Sense](#commonsense) or\n[World Knowledge](#worldknowledge).\n\n## Linguistic Categorization\n\nThe dataset is designed to allow for analyzing many levels of natural language understanding, from\nword meaning and sentence structure to high-level reasoning and application of world knowledge. To\nmake this kind of analysis feasible, we first identify four broad categories of phenomena:\n\n* Lexical Semantics\n* Predicate-Argument Structure\n* Logic\n* Knowledge and Common Sense\n\nHowever, since these are vague and often arguable when it comes to entailment, we divide each\ninto a larger set of fine-grained subcategories. Every example in the dataset is labeled with at\nleast one fine-grained category, and many examples have more than one, often spanning multiple\ncoarse-grained categories as well, since they often inevitably cooccur, especially when using\nnatural sentences. Having a large set of fine-grained categories allows us to ensure a baseline\nlevel of diversity of phenomena in the diagnostic dataset, while also providing a powerful error\nanalysis tool to GLUE users.\n\nDescriptions of all of the fine-grained categories are given below. Bear in mind that these are just\none lens that can be used to understand linguistic phenomena and entailment, and since our\nunderstanding of how language works is incomplete, there is room to argue about how examples should\nbe categorized, what the categories should be, and, for some fine-grained subcategories, which\ncoarse-grained category they should be placed under. These
1categories are not based on any\nparticular linguistic theory, but broadly based on issues that linguists have often identified and\nmodeled in the study of syntax and semantics. When doing error analysis according to any of these\ncategories, please first read their description and look at some examples.\n\n### Lexical Semantics\n\nThese phenomena center on aspects of _word meaning_.\n\n#### Lexical Entailment\n\nEntailment can be applied not only on the sentence level, but the word level. For example, we say\n`dog` lexically entails `animal` because anything that is a dog is also an animal, and `dog`\nlexically contradicts `cat` because it's impossible to be both at once. This applies to all kinds of\nwords (nouns, adjectives, verbs, many prepositions, etc.) and the relationship between lexical and\nsentential entailment has been deeply explored, e.g., in\n[systems](https://nlp.stanford.edu/~wcmac/papers/natlog-wtep07.pdf)\nof [Natural](https://nlp.stanford.edu/~wcmac/papers/natlog-iwcs09.pdf)\n[Logic](https://web.stanford.edu/~icard/logic&language/CSLIworkshop.pdf).\nThis connection often hinges on [_monotonicity_](#monotonicity) in language, so many Lexical Entailment examples will\nalso be tagged with one of the Monotone categories, though we don't do this in every case (see\n[Definite Descriptions and Monotonicity](#definitedescriptionsandmonotonicity)).\n\n\x3c!---\nExample:\n```\nDomain: Wikipedia\nPremise: The villain is the character who tends to have a negative effect on other characters.\nHypothesis: The villain is the character who tends to have a negative impact on other characters.\nForward Label: Entailment\nBackward Label: Entailment\nTags: Lexical Entailment\n```\n---\x3e\n\n#### Morphological Negation\n\nThis is a special case of lexical contradiction where one word is derived from\nthe other: from `affordable` to `unaffordable`, `agree` to `disagree`, etc. We also include examples\nlike `ever` and `never`. We also label these examples with [Negation](#negation) or\n[Double Negation](#doublenegation), since they can be viewed as involving a word-level logical\nnegation.\n\n\x3c!---\nExample:\n```\nDomain: Reddit\nPremise: We built our society on unclean energy.\nHypothesis: We built our society on clean energy.\nForward Label: Contradiction\nBackward Label: Contradiction\nTags: Morphological negation, Negation\n```\n---\x3e\n\n#### Factivity\n\nPropositions appearing in a sentence may be in any entailment relation with the\nsentence as a whole, depending on the context in which they appear. In many cases, this is\ndetermined by lexical triggers (usually verbs or adverbs) in the sentence.\nFor example,\n\n* `I recognize that X` entails `X`\n* `I did not recognize that X` entails `X`\n* `I believe that X` does not entail `X`\n* `I am refusing to do X` contradicts `I am doing X`\n* `I am not refusing to do X` does not contradict `I am doing X`\n* `I almost finished X` contradicts `I finished X`\n* `I barely finished X` entails `I finished X`\n\nConstructions like the one with `recognize` are often called _factive_, since the entailment (of `X`\nabove, regarded as a presupposition) persists even under negation.\nConstructions like the one with `refusing` above are often called _implicative_, and are sensitive\nto negation.\nThere are also cases where a sentence (non-)entails the existence of an entity mentioned in it,\ne.g.,\n\n* `I have found a unicorn` entails `A unicorn exists`\n* `I am looking for a unicorn` doesn't necessarily entail `A unicorn exists`\n\nReadings where the entity does not necessarily exist are often called _intensional readings_, since\nthey seem to deal with the properties denoted by a description (its _intension_) rather than being\nreducible to the set of entities that match the description (its _extension_, which in cases of\nnon-existence will be empty).\n\nWe place all examples involving these phenomena under the label of Factivity.\nWhile it often depends on context to determine whether a nested proposition or existence of an\nentity is entailed by the overall statement, very often it relies heavily on lexical triggers,\nso we place the category under Lexical Semantics.\n\n#### Symmetry
1/Collectivity\n\nSome propositions denote symmetric relations, while others do not; e.g.,\n\n* `John married Gary` entails `Gary married John`\n* `John likes Gary` does not entail `Gary likes John`\n\nFor symmetric relations, they can often be rephrased by collecting both arguments into the subject:\n\n* `John met Gary` entails `John and Gary met`\n\nWhether a relation is symmetric, or admits collecting its arguments into the subject, is often\ndetermined by its head word (e.g., `like`, `marry` or `meet`), so we classify it under Lexical\nSemantics.\n\n#### Redundancy\n\nIf a word can be removed from a sentence without changing its meaning, that means the word's meaning\nwas more-or-less adequately expressed by the sentence; so, identify these cases reflects an\nunderstanding of both lexical and sentential semantics.\n\n\x3c!---\nExample:\n```\nDomain: News\nPremise: Twitch has routinely given away free games and in-game content to Twitch Prime subscribers in the past.\nHypothesis: Twitch has routinely given away games and in-game content to Twitch Prime subscribers in the past.\nForward Label: Entailment\nBackward Label: Entailment\nTags: Redundancy\n```\n---\x3e\n\n#### Named Entities\n\nWords often name entities that exist out in the world. There are many different kinds of\nunderstanding we might wish to understand about these names, including their compositional\nstructure (for example, `the Baltimore Police` is the same as `the Police of the City of Baltimore`)\nor their real-world referents and acronym expansions (for example, `SNL` is `Saturday Night Live`).\nThis category is closely related to [World Knowledge](#worldknowledge), but focuses on the\nsemantics of names as lexical items rather than background knowledge about their denoted entities.\n\n#### Quantifiers\n\nLogical quantification in natural language is often expressed through lexical triggers such as\n`every`, `most`, `some`, and `no`. While we reserve the categories in\n[Quantification](#quantification) and [Monotonicity](#monotonicity) for entailments involving\noperations on these quantifiers and their arguments, we choose to regard the interchangeability of\nquantifiers (e.g., in many cases `most` entails `many`) as a question of lexical semantics.\n\n### Predicate-Argument Structure\n\nAn important component of understanding the meaning of a sentence is understanding how its parts are\ncomposed together into a whole. In this category, we address issues across that spectrum, from\nsyntactic ambiguity to semantic roles and coreference.\n\n#### Syntactic Ambiguity: Relative Clauses, Coordination Scope\n\nThese two categories deal purely with resolving syntactic ambiguity. Relative clauses and\ncoordination scope are both sources of a great amount of ambiguity in English.\n\n#### Prepositional phrases\n\nPrepositional phrase attachment is a particularly difficult problem that syntactic parsers in NLP\nsystems continue to struggle with. We view it as a problem both of syntax and semantics, since\nprepositional phrases can express a wide variety of semantic roles and often semantically apply\nbeyond their direct syntactic attachment.\n\n#### Core Arguments\n\nVerbs select for particular arguments, especially as their subject and object, which might be\ninterchangeable depending on the context or the surface form. One example is the\n_ergative alternation_:\n\n* `Jake broke the vase` entails `the vase broke`.\n* `Jake broke the vase` does not entail `Jake broke`.\n\nOther rearrangements of core arguments, such as those seen in\n[Symmetry/Collectivity](#symmetrycollectivity), also fall under the Core Arguments label.\n\n#### Alternations: Active/Passive, Genitives/Partitives, Nominalization, Datives\n\nAll four of these categories correspond to _syntactic alternations_ that are known to follow\nspecific patterns in English:\n\n* Active/Passive: `I saw him` is equivalent to `He was seen by me` and entails `He was seen`.\n* Genitives/Partitives: `the elephant's foot` is the same thing as `the foot of the elephant`.\n* Nominalization: `I caused him to submit his resignation` entails `I caused the submission of his resignation`.\n* Datives: `I baked him a cake` entails `I baked a cake for him` and `I baked a cake` but not `I baked him`.\n\n#### Ellipsis/Implicits\n\nOften, the argument of a verb or other predicate is omitted (_elided_) in the text, with the reader\nfilling in the gap. We can construct entailment examples by explicitly filling in the gap with\nthe correct or incorrect referents.\n\nFor example:\n* Premise: `Putin is so entrenched within Russia\u2019s ruling system that many of its members can imagine no other leader.`\n* Entails: `Putin is so entrenched within Russia\u2019s ruling system that many of its members can imagine no other leader than Putin.`\
1n* Contradicts: `Putin is so entrenched within Russia\u2019s ruling system that many of its members can imagine no other leader than themselves.`\n\nThis is often regarded as a special case of _anaphora_, but we decided to split out these cases from\nexplicit anaphora, which is often also regarded as a case of coreference (and attempted to some\ndegree in modern coreference resolution systems).\n\n#### Anaphora/Coreference\n\n_Coreference_ refers to when multiple expressions refer to the same entity or event. It is closely\nrelated to _Anaphora_, where the meaning of an expression depends on another (antecedent) expression\nin context. These phenomena have significant overlap, for example, with pronouns (`she`, `we`, `it`),\nwhich are anaphors that are co-referent with their antecedents. However, they also may occur\nindependently, for example, coreference between two definite noun phrases (e.g., `Theresa May` and\n`the British Prime Minister`) that refer to the same entity, or anaphora from a word like `other`\nwhich requires an antecedent to distinguish something from. In this category we only include cases\nwhere there is an explicit phrase (anaphoric or not) that is co-referent with an antecedent or other\nphrase.\n\nWe construct examples for these in much the same way as for [Ellipsis/Implicits](#ellipsisimplicits).\n\n#### Intersectivity\n\nMany modifiers, especially adjectives, allow _non-intersective_ uses, which affect their entailment\nbehavior. For example:\n\n* Intersective: `He is a violinist and an old surgeon` entails `He is an old violinist` and `He is a surgeon`\n* Non-intersective: `He is a violinist and a skilled surgeon` does _not_ entail `He is a skilled violinist`\n* Non-intersective: `He is a fake surgeon` does _not_ entail `He is a surgeon`\n\nGenerally, an _intersective_ use of a modifier, like `old` in `old men`, is one which may be\ninterpreted as referring to the set of entities with both properties (they are `old` and they are\n`men`). Linguists often formalize this using set intersection, hence the name.\n\nIt is related to [Factivity](#factivity); for example `fake` may be regarded as a\ncounter-implicative modifier, and these examples will be labeled as such. However, we choose to\ncategorize intersectivity under predicate-argument structure rather than lexical semantics, because\ngenerally the same word will admit both intersective and non-intersective uses, so it may be\nregarded as an ambiguity of argument structure.\n\n#### Restrictivity\n\n`Restrictivity` is most often used to refer to a property of uses of noun modifiers; in particular,\na _restrictive_ use of a modifier is one that serves to identify the entity or entities being\ndescribed, whereas a _non-restrictive_ use adds extra details to the identified entity. The\ndistinction can often be highlighted by entailments:\n\n* Restrictive: `I finished all of my homework due today` does not entail `I finished all of my homework`\n* Non-restrictive: `I got rid of all those pesky bedbugs` entails `I got rid of all those bedbugs`.\n\nModifiers that are commonly used non-restrictively are appositives, relative clauses starting with\n`which` or `who` (although these _can_ be restrictive, despite what your English teacher might tell\nyou), and expletives (e.g. `pesky`). However, non-restrictive uses can appear in many forms.\n\nAmbiguity in restrictivity is often employed in certain kinds of [jokes](https://xkcd.com/90/)\n(warning: language).\n\n### Logic\n\nOnce you understand the structure of a sentence, there is often a baseline set of shallow\nconclusions you can draw using logical operators.\n\nThere is a long tradition of modeling natural language semantics using the mathematical tools of\nlogic. Indeed, the development of mathematical logic was initially by questions about natural\nlanguage meaning, from Aristotelian syllogisms to Fregean symbols. The notion of entailment\nis also borrowed from mathematical logic. So it is no surprise that logic plays an important role\nin natural language inference.\n\n#### Propositional Structure: Negation, Double Negation, Conjunction, Disjunction, Conditionals\n\nAll of the basic operations of propositional logic appear in natural language, and we tag them where\nthey are relevant to our examples:\n\n* Negation: `The cat sat on the mat` contradicts `The cat did not sit on the mat`.\n* Double negation: `The market is not impossible to navigate` entails `The market is possible to navigate`.\n* Conjunction: `Temperature and snow consistency must be just right` entails `Temperature must be just right`.\n* Disjunction: `Life is either a daring adventure or nothing at all` does not entail, but is entailed by, `Life is a daring adventure`.\n* Conditionals: `If both apply, they are essentially impossible` does not entail `They are essentially impossible`.\n\nConditionals are a little bit more complicated because their use in language does not always mirror\ntheir meaning in logic. For example, they may be used at a higher level than the at-issue assertion:\n\n* `If you think about it, it's the perfect reverse psych
1ology tactic` entails `It's the perfect reverse psychology tactic`\n\n#### Quantification: Universal, Existential\n\nQuantifiers are often triggered by words such as `all`, `some`, `many`, and `no`. There is a rich\nbody of work modeling their meaning in mathematical logic with generalized quantifiers. In these two\ncategories, we focus on straightforward inferences from the natural language analogs of\nuniversal and existential quantification:\n\n* Universal: `All parakeets have two wings` entails, but is not entailed by `My parakeet has two wings`.\n* Existential: `Some parakeets have two wings` does not entail, but is entailed by `My parakeet has two wings`.\n\n#### Monotonicity: Upward Monotone, Downward Monotone, Non-Monotone\n\nMonotonicity is a property of argument positions in certain logical systems. In general, it gives a\nway of deriving entailment relations between expressions that differ on only one subexpression. In\nlanguage, it can explain how some entailments propagate through logical operators and quantifiers.\n\nFor example, note that `pet` entails `pet squirrel`, which further entails `happy pet squirrel`.\nWe can demonstrate how the quantifiers `a`, `no` and `exactly one` differ with respect to\nmonotonicity:\n\n* `I have a pet squirrel` entails `I have a pet`, but not `I have a happy pet squirrel`.\n* `I have no pet squirrels` does not entail `I have no pets`, but does entail `I have no happy pet squirrels`.\n* `I have exactly one pet squirrel` entails neither `I have exactly one pet` nor `I have exactly one happy pet squirrel`.\n\nIn all of these examples, the pet squirrel appears in what we call the _restrictor_ position of the\nquantifier. We say:\n\n* `a` is _upward monotone_ in its restrictor: an entailment in the restrictor yields an entailment of the whole statement.\n* `no` is _downward monotone_ in its restrictor: an entailment in the restrictor yields an entailment of the whole statement _in the opposite direction_.\n* `exactly one` is _non-monotone_ in its restrictor: entailments in the restrictor do not yield entailments of the whole statement.\n\nIn this way, entailments between sentences that are built off of entailments of sub-phrases almost\nalways rely on monotonicity judgments; see, for example, [Lexical Entailment](#lexicalentailment).\nHowever, because this is such a general class of sentence pairs, to keep the Logic category\nmeaningful we do not always tag these examples with monotonicity; see\n[Definite Descriptions and Monotonicity](#definitedescriptionsandmonotonicity) for details.\n\nTo draw an analogy, these types of monotonicity are closely related to\n[covariance, contravariance, and invariance](https://en.wikipedia.org/wiki/Covariance_and_contravariance_(computer_science))\nof type arguments in programming languages with subtyping.\n\n#### Richer Logical Structure: Intervals/Numbers, Temporal\n\nThere are some higher-level facets of reasoning that have been traditionally modeled using logic;\nthese include actual mathematical reasoning (entailments based off of numbers) and temporal\nreasoning (which is often modeled as reasoning about a mathematical timeline).\n\n* Intervals/Numbers: `I have had more than 2 drinks tonight` entails `I have had more than 1 drink tonight`.\n* Temporal: `Mary left before John entered` entails `John entered after Mary left`.\n\n### Knowledge & Common Sense\n\nStrictly speaking, world knowledge and common sense are required on every level of language\nunderstanding, for disambiguating word senses, syntactic structures, anaphora, and more. So our\nentire suite (and any test of entailment) does test these features to some degree. However, in these\ncategories, we gather examples where the entailment rests not only on correct disambiguation of the\nsentences, but also application of extra knowledge, whether it is concrete knowledge about world\naffairs or more common-sense knowledge about word meanings or social or physical dynamics.\n\n#### World Knowledge\n\nIn this category we focus on knowledge that can clearly be expressed as facts, as well as broader\nand less common geographical, legal, political, technical, or cultural knowledge. Examples:\n\n* `This is the most oniony article I've seen on the entire internet` entails `This article reads like satire`.\n* `The reaction was strongly exothermic` entails `The reaction media got very hot`.\n* `There are amazing hikes around Mt. Fuji` entails `There are amazing hikes in Japan` but not `There are amazing hikes in Nepal`.\n\n#### Common Sense\n\nIn this category we focus on knowledge that is more difficult to express as facts and that we expect\nto be possessed by most people independent of cultural or educational background. This includes\na basic understanding of physical and social dynamics as well as lexical meaning (beyond simple\nlexical entailment or logical relations). Examples:\n\n* `The announcement of Tillerson\u2019s departure sent shock waves across the globe` contradicts `People across the globe were prepared for Tillerson's departure`.\n* `Marc Sims has been seeing his barber once a week, for several years` entails `Marc Sims has been getting his hair cut once a week, for several years.`\n* `Hummingbirds are really attracted to bright orange and red (hence why the feeders are usually these colours)` entails `The feeders are usually coloured so as to attract hummingbirds`.\n"},260:function(e,t,n){e.exports=n(439)},424:function(e){e.exports={faqs:[{question:"How do I use GLUE?",answer:["To evaluate on GLUE, collect your system's predictions on the nine primary tasks and the one auxiliary task.","<ul>","<li>Get data for all tasks (with the exception of MRPC, see below) from the 'Tasks' section. For the auxiliary task, you only need the TSV from 'Diagnostic Main'. Each primary task comes as a zip folder containin a standardized train, dev, and (unlabeled) test split.</li>","<li>Use the IDs and labels present in the unlabeled test TSVs to generate one TSV of predictions for each of the eleven TSVs (separate test split for MNLI matched and mismatched), where each TSV has a header and each line follows the format 'id [TAB] label'.</li>","<li>Make sure that each prediction TSV is named according to the following:</li>","<ul>","<li>Corpus of Linguistic Acceptability: CoLA.tsv</li>","<li>Stanford Sentiment Treebank: SST-2.tsv</li>","<li>Microsoft Research Paraphrase Corpus: MRPC.tsv</li>","<li>Semantic Textual Similarity Benchmark: STS-B.tsv</li>","<li>Quora Question Pairs: QQP.tsv</li>","<li>MultiNLI Matched: MNLI-m.tsv</li>","<li>MultiNLI Mismatched: MNLI-mm.tsv</li>","<li>Question NLI: QNLI.tsv</li>","<li>Recognizing Textual Entailment: RTE.tsv</li>","<li>Winograd NLI: WNLI.tsv</li>","<li>
1Diagnostic: AX.tsv</li>","</ul>","<li>Create a zip of the prediction TSVs, without any subfolders, e.g. using 'zip -r submission.zip *.tsv'.</li>","<li>Upload this zip using the 'Submit' section, filling in details of the method used to generate the predictions.</li>","</ul>","You may upload at most two submissions a day.","A sample submission with the necessary formatting is available <a href='https://gluebenchmark.com/assets/CBOW.zip'>here</a>.","You can use <a href='https://github.com/nyu-mll/GLUE-baselines'>the code for the baselines</a> as a starting point.","See below if you are running into issues submitting."]},{question:"Are there any rules or restrictions on submitted systems?",answer:["Our only firm rule is that you may not train or tune your systems on the test sets for the nine primary tasks. This includes the test sets for which no labels are available to the public \u2014 you may not label these yourself or use them in any other way to improve your submitted system. In addition, you may not publish any kind of detailed analysis of the test sets for the nine primary tasks. Systems associated with such work will be removed from the leaderboard.<br /><br />","Beyond this, you may submit results from any kind of system that is capable of producing labels for the nine target tasks and the analysis tasks. This includes systems that do not share any components across tasks or systems not based on machine learning."]},{question:"Is there a deadline?",answer:"No. GLUE is an open-ended competition with no deadline or set end date. However, performance on GLUE is saturated, and in response we have released a harder benchmark, <a href='https://super.gluebenchmark.com/'>SuperGLUE</a>. SuperGLUE includes more challenging tasks, greater task diversity, and other quality-of-life updates. We recommend evaluating on SuperGLUE."},{question:"Where can I get the data for the tasks?",answer:["You can download per-task training, development, and unlabeled test data from the 'Tasks' section.","We also provide a convenience <a href='https://gist.github.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e'>helper script</a> to download all of the data.","Some quirks about the data:<br/>","We are not able to distribute MRPC, but you can download the data from the <a href='https://www.microsoft.com/en-us/download/details.aspx?id=52398'>original site</a> and use our script to split the training data according to our standardized splits and format the test data.<br/>","Similarly, for SST, the data provided is already tokenized. We're working on obtaining a version that is not tokenized. Feel free to train on other distributions of SST, as long as they use the standard train/dev/test split.<br/>","In the MNLI data we provide, column 11 of the train data is the gold label, whereas column *15* of the dev files is the gold label.<br/>","For QQP, the training and dev files contain some malformed examples. For our experiments, we filter out those examples, but you're welcome to use other distributions of the data for training.<br/>","For QNLI, we discovered an artifact in the originally released data that allowed the task to be modeled as an easier problem that intended. We've re-released the data, so be sure to re-download.<br/>","Original QNLIv1 data can be found at <a href='https://firebasestorage.googleapis.com/v0/b/mtl-sentence-representations.appspot.com/o/data%2FQNLI.zip?alt=media&token=c24cad61-f2df-4f04-9ab6-aa576fa829d0'>this link</a>."]},{question:"What license is the GLUE data distributed under?",answer:"The primary GLUE tasks are built on and derived from existing datasets. We refer users to the original licenses accompanying each dataset."},{question:"Where do the datasets come from?",answer:["<a href='https://openreview.net/pdf?id=rJ4km2R5t7'>Our paper</a> contains details and citations for all of the data used and the processing applied to them.","We thank <a href='https://www.quora.com'>Quora</a> for sharing additional data for QQP.","We also thank Ernie Davis for providing data from the Winograd Schema Challenge and Alex Warstadt and colleagues for sharing data for <a href='nyu-mll.github.io/CoLA/'>CoLA</a>."]},{question:"Why is my submission not appearing on leaderboard?",answer:["If you have just submitted, please wait at least 5 minutes for grader to run and grade your submission.","<br/>","First, in your profile, check if submission is present. If submission's status is error, hover over the error symbol to see it.","<br/>","If you don't see it in the leaderboard, wait for 24 hours to see the submission on the leaderboard.","<br/>","If you have multiple submissions, all submissions should be present in your profile but only the top-scoring one would be shown on the leaderboard.","<br/>","If your submission's score is lower than the best baseline in the GLUE manuscript (70%), it will not be displayed on the leaderboard as well.","<br/>","In other cases, when your submission is still not present on the leaderboard, check below or contact us.","A submission may not be graded in case of any of the following issues:","<ol>","<li>If the top level directory of the zip does not contain a file for all eleven tasks.</li>","<li>If you are missing any example ID for any task. The IDs for task in TSVs are incremental and start from 0. Make sure to use same IDs as in the test TSVs.</li>","</ol>","Make sure to upload results for diagnostics dataset as AX.tsv","We use direct string matching for matching the labels.","<ol>","<li>RTE and QNLI: Labels should be either 'entailment' or 'not_entailment'</li>","<li>MNLI and AX: Labels should be either 'entailment', 'neutral' or 'contradiction'</li>","<li>For all other tasks, the labels should match the training data labels.</li>","</ol>","For validating whether you contain all of the IDs and each of task, run 'wc -l *.tsv' in you submission folder. You should see:","<pre>"," 1105 AX.tsv<br/> 1064 CoLA.tsv<br/> 9848 MNLI-mm.tsv<br/> 9797 MNLI-m.tsv<br/> 1726 MRPC.tsv<br/> 5464 QNLI.tsv<br/> 390966 QQP.tsv<br/> 3001 RTE.tsv<br/> 1822 SST-2.tsv<br/> 1380 STS-B.tsv<br/> 147 WNLI.tsv<br/> 426597 total","</pre>","In your output for above command, TSV names and counts should match exactly.","If you originally evaluated on the previous version of QNLI (pre-01/30/19) and have not yet submitted a test submission on the new version, then we have masked out the old result and will update once we have received your new submission."]},{question:"What information do you need about my submission?",answer:["We ask for seven pieces of information:","<ol>","<li>A short name for your system, which will be displayed in the leaderboard.</li>","<li>A URL for a paper or (if one is not available) website or code repository describing your system.</li>","<li>
1A sentence or two describing your system. Make sure to mention any outside data or resources you use.</li>","<li>A sentence or two explaining how you share parameters across tasks (or stating that you don't share parameters).</li>","<li>The total number of trained parameters in your model. Do not count word or word-part embedding parameters, even if they are trained.</li>","<li>The total number of trained parameters in your model that are shared across multiple tasks. If some parameters are shared across some but not all of your tasks, count those. Do not count word or word-part embedding parameters, even if they are trained.</li>","<li>Whether you want your submission to be visible on the public leaderboard.</li>","</ol>"]},{question:"Can I make a public submission to the leaderboard anonymously?",answer:["You are welcome to submit to the leaderboard without using your real name <b>as long as</b> you include a link to a paper describing your submission. This will often involve anonymous paper archives like <a href='https://openreview.net/group?id=OpenReview.net/Anonymous_Preprint'>OpenReview Anonymous Preprint</a>. Currently, we always show the name that is associated with your Google account, so in order to submit anonymously, you will need to create an anonymous Google account. We are working on a workaround to make this easier."]},{question:"How does the leaderboard work?",answer:["We calculate scores each of the tasks based on their individual metrics. All metrics are scaled by 100x (i.e., as percentages). These scores are then averaged to get the final score. For tasks with multiple metrics (including MNLI), the metrics are averaged.","On the leaderboard, only the top scoring submission of a user is shown or ranked by default. Other submissions can be viewed under the expanded view for each user.","Competitors may submit privately, preventing their results from appearing.","The leaderboard is updated every 24 hours and we don't show submissions that have a score below 70% even if they are public.","Note: MNLI matched and mismatched are considered one task for purpose of scoring."]},{question:"How can I see more information about a submission?",answer:["On the leaderboard, click on a submission. A drawer will appear from the side which will provide more information about a submission.","On the side drawer, click on 'More Information' to go to an expanded submission details page."]},{question:"I get weird results for QQP or WNLI. What gives?",answer:["QQP: There is a difference in the dev and test distributions that likely explains discrepancies observed between scores for the two.","WNLI: The train/dev split for WNLI is correct, but turns out to be somewhat adversarial: when two examples contain the same sentence, that usually means they'll have opposite labels. The train and dev splits may share sentences, so if a model has overfit the training set, it may get worse than chance accuracy on WNLI on the dev set. Additionally, the test set has a different label distribution than the train and dev sets."]},{question:"Do you have a discussion group?",answer:"Yes, join <a href='https://groups.google.com/forum/#!forum/glue-benchmark-discuss'>here</a>."},{question:"How should I cite GLUE?",answer:["If you use GLUE, please cite all the datasets you use. In addition, we encourage you to use the following BibTeX citation for GLUE itself:","<pre style='white-space: pre-wrap;'>","<code>","@inproceedings{wang2019glue,<br/>"," title={{GLUE}: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding},<br/>"," author={Wang, Alex and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel R.},<br/>"," note={In the Proceedings of ICLR.},<br/>"," year={2019}<br/>","}<br/>","</code>","</pre>","If you evaluate using GLUE, we also highly recommend citing the papers that originally introduced the nine GLUE tasks, both to give the original authors their due credit and because venues will expect papers to describe the data they evaluate on. The following provides BibTeX for all of the GLUE tasks, except QQP, for which we recommend adding a footnote to <a href='https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs'>this page</a>.","<pre style='white-space: pre-wrap;'>","<code>","@article{warstadt2018neural,<br/>"," title={Neural Network Acceptability Judgments},<br/>"," author={Warstadt, Alex and Singh, Amanpreet and Bowman, Samuel R.},<br/>"," journal={arXiv preprint 1805.12471},<br/>"," year={2018}<br/>","}<br/>","@inproceedings{socher2013recursive,<br/>"," title={Recursive deep models for semantic compositionality over a sentiment treebank},<br/>"," author={Socher, Richard and Perelygin, Alex and Wu, Jean and Chuang, Jason and Manning, Christopher D and Ng, Andrew and Potts, Christopher},<br/>"," booktitle={Proceedings of EMNLP},<br/>"," pages={1631--1642},<br/>"," year={2013}<br/>","}<br/>","@inproceedings{dolan2005automatically,<br/>"," title={Automatically constructing a corpus of sentential paraphrases},<br/>"," author={Dolan, William B and Brockett, Chris},<br/>"," booktitle={Proceedings of the International Workshop on Paraphrasing},<br/>"," year={2005}<br/>","}<br/>","@book{agirre2007semantic,<br/>"," editor = {Agirre, Eneko and M`arquez, Llu'{i}s and Wicentowski, Richard},<br/>"," title = {Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007)},<br/>"," month = {June},<br/>"," year = {2007},<br/>"," address = {Prague, Czech Republic},<br/>"," publisher = {Association for Computational Linguistics},<br/>","}<br/>","@inproceedings{williams2018broad,<br/>"," author = {Williams, Adina and Nangia, Nikita and Bowman, Samuel R.},<br/>"," title = {A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference},<br/>"," booktitle = {Proceedings of NAACL-HLT},<br/>"," year = 2018<br/>","}<br/>","@inproceedings{rajpurkar2016squad,<br/>"," author = {Rajpurkar, Pranav and Zhang, Jian and Lopyrev, Konstantin and Liang, Percy}<br/>"," title = {{SQ}u{AD}: 100,000+ Questions for Machine Comprehension of Text},<br/>"," booktitle = {Proceedings of EMNLP}<br/>"," year = {2016},<br/>"," publisher = {Association for Computational Linguistics},<br/>"," pages = {2383--2392},<br/>"," location = {Austin, Texas},<br/>","}<br/>","@incollection{dagan2006pascal,<br/>"," title={The {PASCAL} recognising textual entailment challenge},<br/>"," author={Dagan, Ido and Glickman, Oren and Magnini, Bernardo},<br/>"," booktitle={Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment},<br/>"," pages={177--190},<br/>"," year={2006},<br/>"," publisher={Springer}<br/>","}<br/>","@article{bar2006second,<br/>"," title={The second {PASCAL} recognising textual entailment challenge},<br/>"," author={Bar Haim, Roy and Dagan, Ido and Dolan, Bill and Ferro, Lisa and Giampiccolo, Danilo and Magnini, Bernardo and Szpektor, Idan},<br/>"," year={2006}<br/>","}<br/>","@inproceedings{giampiccolo2007third,<br/>"," title={The third {PASCAL} recognizing textual entailment challenge},<br/>"," author={Giampiccolo, Danilo and Magnini, Bernardo and Dagan, Ido and Dolan, Bill},<br/>"," booktitle={Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing},<br/>"," pages={1--9},<br/>"," year={2007},<br/>"," organization={Association for Computational Linguistics},<br/>","}<br/>","@article{bentivogli2009fifth,<br/>"," title={The Fifth {PASCAL} Recognizing Textual Entailment Challenge},<br/>"," author={Bentivogli, Luisa and Dagan, Ido and Dang, Hoa Trang and Giampiccolo, Danilo and Magnini, Bernardo},<br/>"," booktitle={TAC},<br/>"," year={2009}<br/>","}<br/>","@inproceedings{levesque2011winograd,<br/>"," title={The {W}inograd schema challenge},<br/>"," author={Levesque, Hector J and Davis, Ernest and Morgenstern, Leora},<br/>"," booktitle={{AAAI} Spring Symposium: Logical Formalizations of Commonsense Reasoning},<br/>"," volume={46},<br/>"," pages={47},<br/>"," year={2011}<br/>","}<br/>","</code>","</pre>"]},{question:"I have a suggestion, I found a bug, or I just want to chat. How do I reach you?",answer:"Contact us: glue-benchmark-admin [at] googlegroups [dot] com"}]}},427:function(e,t,n){},430:function(e,t,n){e.exports=n.p+"static/media/ml2_logo.e6a3cc81.png"},431:function(e,t,n){e.exports=n.p+"static/media/uw_nlp_logo.27d7c248.png"},432:function(e,t,n){e.exports=n.p+"static/media/deepmind_logo.e5723542.png"},437:function(e,t,n){},439:function(e,t,n){"use strict";n.r(t);var a=n(0),r=n(36),o=n(31),i=n(30),s="LOGOUT_USER",l="USER_HAS_LOGGED_IN",c="USER_HAS_LOGGED_OUT",u="LOGIN_WITH_GOOGLE",m="TASKS_REQUESTED",d="TASKS_FETCHED",p="RESULTS_REQUESTED",h="RESULTS_FETCHED",f="START_CAN_UPLOAD_CHECK",g="FINISH_CAN_UPLOAD_CHECK",b="UPDATE_SELECTED_USER",y="START_FETCH_USER_SUBMISSION",v="FINISH_FETCH_USER_SUBMISSION",w="START_FETCH_SUBMISSION",E="FINISH_FETCH_SUBMISSION",k="CHANGE_THEME",T={MTL_PROJECT_NAME:"mtl-sentence-representations",PROJECT_TITLE:"GLUE",AUTHENTICATION_LOCAL_STORAGE_KEY:"MTL_SENTENCE_REPRESENTATIONS_LOCAL_STORAGE_$UID",SELECT_TSV_LABEL:"Select Test TSV with labels",SELECT_ZIP_LABEL:"Select Zip",THEME:{DARK:{WHITE:"#fff",TONED_WHITE:"#aaa",BLACK:"#000",RED:"#bd492a",YELLOW:"#efdab9",TABS_TEXT_COLOR:"#efdab9",DARK_YELLOW:"#e2ad4b",TABS_SELECTED_TEXT_COLOR:"#e2ad4b",GREEN:"#8bbbae",RAISED_BUTTON_COLOR:"#8bbbae",BLUE:"#64a2c8",ACCENT_1_COLOR:"#64a2c8",GREY:"#5c5a5b",ORANGE:"#d07136",FADE_GREY_BORDER:"#494647",FADE_GREY:"#393738",TABS_COLOR:"#393738",BACKGROUND_COLOR:"#343233",BODY_BACKGROUND:"#343233",GOOGLE_BUTTON_COLOR:"#fff"},LIGHT:{WHITE:"#fff",TONED_WHITE:"#aaa",BLACK:"#2b5283",RAISED_BUTTON_COLOR:"#2b5283",GOOGLE_BUTTON_COLOR:"#fff",RED:"#bd492a",ACCENT_1_COLOR:"#bd492a",YELLOW:"#222",TABS_TEXT_COLOR:"#aaa",TABS_SELECTED_TEXT_COLOR:"#fff",DARK_YELLOW:"#fff",GREEN:"#fff",BLUE:"#2b5283",GREY:"#fff",ORANGE:"#d07136",FADE_GREY_BORDER:"#205493",FADE_GREY:"#fff",TABS_COLOR:"#004C97",BACKGROUND_COLOR:"#2b5283",BODY_BACKGROUND:"#eee"}}},S=n(48),I=n(142),L=S.a.initializeApp(I),_=L.database(),x=L.auth(),N=L.storage(),C=(T.AUTHENTICATION_LOCAL_STORAGE_KEY,new S.a.auth.GoogleAuthProvider),O=function(){return(O=Object.assign||function(e){for(var t,n=1,a=arguments.length;n<a;n++)for(var r in t=arguments[n])Object.prototype.hasOwnProperty.call(t,r)&&(e[r]=t[r]);return e}).apply(this,arguments)};function A(e,t){void 0===t&&(t=!1);var n={};for(var a in e)if(e.hasOwnProperty(a)){var r=e[a];if(!r)continue;if(!r.successful)continue;n[r.uploaderUid]?n[r.uploaderUid].allSubmissions.push(r):n[r.uploaderUid]={allSubmissions:[r],displayName:"",avatarImageUrl:""}}
1var o=[];return Object.keys(n).forEach(function(e){var t=_.ref().child("users/"+e+"/publicFields").once("value"),n=new Promise(function(n,a){t.then(function(t){return n({value:t,userId:e})},function(e){return a(e)})});o.push(n)}),Promise.all(o).then(function(e){return e.filter(function(e){return!(e instanceof Error)})}).then(function(e){return e.forEach(function(e,t){var a=e.value.val();a||(console.log(a),a={displayName:"",avatarImageUrl:""}),n[e.userId]=O(O({},n[e.userId]),{displayName:a.displayName,avatarImageUrl:a.avatarImageUrl})}),function(e){var t=[];return Object.keys(e).forEach(function(n){var a=e[n],r=a.allSubmissions;r.sort(R);var o=r[0];r.shift();var i={displayName:a.displayName,avatarImageUrl:a.avatarImageUrl,maxScore:o.macroScore,maxScoreSubmission:o,otherSubmissions:r};t.push(i)}),t.sort(P),t}(n)})}function R(e,t){return"-"===e.macroScore.toString()?1:"-"===t.macroScore.toString()?-1:e.macroScore>t.macroScore?-1:e.macroScore<t.macroScore?1:0}function P(e,t){return"-"===e.maxScore.toString()?1:"-"===t.maxScore.toString()?-1:e.maxScore>t.maxScore?-1:e.maxScore<t.maxScore?1:0}function U(e){if(e){var t=_.ref("users").child(e.uid),n=[t.child("publicFields").once("value"),t.child("privateFields").once("value"),t.child("isAdmin").once("value"),t.child("isBanned").once("value")];return function(t){return Promise.all(n).then(function(n){var a={publicFields:n[0].val()||{avatarImageUrl:e.photoURL,displayName:e.displayName},privateFields:n[1].val()||{email:e.email,createdOn:Date.now(),googleId:e.uid},isAdmin:n[2].val()||!1,isBanned:n[3].val()||!1,publicButAddable:{}};t(function(e){return{type:l,data:{user:e}}}(a))})}}return{type:c}}function D(e){return function(t){return function(){var e=x.currentUser;if(!e)return Promise.resolve([]);var t=_.ref("users"),n={privateFields:{email:e.email,createdOn:S.a.database.ServerValue.TIMESTAMP,googleId:e.uid},publicFields:{avatarImageUrl:e.photoURL||"https://upload.wikimedia.org/wikipedia/commons/thumb/1/12/User_icon_2.svg/220px-User_icon_2.svg.png",displayName:e.displayName||e.email}},a=t.child(e.uid);return a.once("value").then(function(e){if(null!==e.val()&&e.val().privateFields.email)return Promise.resolve([]);var t=[];return t.push(a.child("privateFields").set(n.privateFields)),t.push(a.child("publicFields").set(n.publicFields)),Promise.all(t)})}().then(function(){t({type:u,data:{result:e}})})}}function j(){return function(e){return x.signOut().then(function(){e({type:s})})}}function M(){return function(e){return e({type:m}),_.ref("tasks").once("value").then(function(t){var n;e((n=t.val(),{type:d,data:{result:n||{}}}))},function(e){console.log(e)})}}function B(e){return{type:g,data:e}}var G=864e5;function q(){return function(e){if(e({type:f}),x.currentUser)return _.ref().child("users/"+x.currentUser.uid+"/publicButAddable/submissions").orderByChild("successful").equalTo(!0).limitToLast(2).once("value").then(function(t){var n=t.val();if(!n||Object.keys(n).length<=1)e(B(!0));else{var a=n[Object.keys(n)[0]];Date.now()-a.createdOn>G?e(B(!0)):e(B(!1))}},function(t){console.log(t),e(B(!1))});e(B(!1))}}function F(e){return{type:h,data:{result:e}}}function W(e){return void 0===e&&(e=!1),function(t){return t({type:p}),e?_.ref("results").orderByChild("macroScore").limitToLast(400).once("value").then(function(e){var t=Object.keys(e.val()).reverse().map(function(e){return new Promise(function(t,n){_.ref("results/"+e).once("value").then(function(n){t({val:n,id:e})},function(e){n(e)})})});return Promise.all(t)}).then(function(n){var a={};return n.forEach(function(t){t&&(e&&Object.keys(a).length>50||(e?!0===t.val.val().successful&&(a[t.id]=t.val.val(),a[t.id].submissionId=t.id):(a[t.id]=t.val.val(),a[t.id].submissionId=t.id)))}),A(a,e).then(function(e){t(F(e))})}):_.ref("leaderboard").child("glue").orderByKey().limitToLast(1).once("value").then(function(e){var n=Object.values(e.val())[0].filter(function(e){return!("-"!==e.maxScore.toString()&&100*e.maxScore<70)});t(F(n))})}}function H(e){return function(t){return t({type:y}),function(e){return _.ref().child("users/"+e+"/publicButAddable/submissions").once("value").then(function(e){var t=e.val(),n=[];return t?(Object.keys(t).forEach(function(e){var t=_.ref().child("results/"+e).once("value"),a=new Promise(function(n,a){t.then(function(t){return n({value:t,submissionId:e})},function(e){return a(e)})});n.push(a)}),Promise.all(n)):[]}).then(function(e){return e.filter(function(e){return!(e instanceof Error)})}).then(function(e){var t={};return e.forEach(function(e,n){var a=e.value.val();a&&(t[e.submissionId]=a)}),t})}(e).then(function(e){t({type:v,data:e})})}}function z(e){return function(t){return t({type:w}),_.ref("results/"+e).once("value").then(function(n){var a;t((a={data:n.val(),id:e},{type:E,data:a}))})}}var Q,V=n(35),J=n(24),Y=n.n(J),K=n(53),X=n.n(K),Z=(Q=function(e,t){return(Q=Object.
1setPrototypeOf||{__proto__:[]}instanceof Array&&function(e,t){e.__proto__=t}||function(e,t){for(var n in t)t.hasOwnProperty(n)&&(e[n]=t[n])})(e,t)},function(e,t){function n(){this.constructor=e}Q(e,t),e.prototype=null===t?Object.create(t):(n.prototype=t.prototype,new n)}),$=n(50),ee=function(e){function t(){return null!==e&&e.apply(this,arguments)||this}return Z(t,e),t.prototype.loginUserWithGoogle=function(){x.signInWithPopup(C)},t.prototype.componentDidMount=function(){var e=this;x.onAuthStateChanged(function(t){e.props.dispatchFinishLoginWithGoogle(t)})},t.prototype.render=function(){var e=(this.props.location.state||{from:"/"}).from;return this.props.userInfo.isAuthenticated&&this.props.userInfo.isBanned?a.createElement("div",null,a.createElement("div",null,"Unfortunately, you have been banned from GLUE and SuperGLUE benchmarks."),a.createElement("div",null,"Contact admins at glue-benchmark-admin [at] googlegroups [dot] com for more information.")):this.props.userInfo.isAuthenticated&&!this.props.userInfo.redirectToReferrer?a.createElement(V.a,{to:"/"}):this.props.userInfo.isCheckingLogin?a.createElement("div",{className:"centered-container"},a.createElement("img",{src:$,className:"App-logo",alt:"Loading"}),";"):a.createElement("div",{className:"centered-container"},this.props.userInfo.redirectToReferrer&&a.createElement(V.a,{to:e||"/"}),a.createElement(Y.a,{variant:"contained",color:"secondary",style:{color:"#fff"},onClick:this.loginUserWithGoogle},a.createElement(X.a,{style:{marginRight:"0.5em"},className:"fa fa-google"}),"Sign in with Google"))},t}(a.Component),te=Object(i.b)(function(e){return{userInfo:e.userInfo,general:e.general}},function(e){return{dispatchFinishLoginWithGoogle:function(t){e(D(t))}}})(ee),ne=n(33),ae=n(29),re={isAuthenticated:!!x.currentUser,uid:x.currentUser?x.currentUser.uid:null,redirectToReferrer:!1,displayName:x.currentUser?x.currentUser.displayName:null,email:x.currentUser?x.currentUser.email:null,avatarImageUrl:x.currentUser?x.currentUser.photoURL:null,isAdmin:!1,isBanned:!1,isCheckingLogin:!0,isFetchingSubmissions:!1,submissions:{},otherSubmissions:{},isSiteReady:!0},oe={isRequested:!1,tasks:{}},ie={isChecking:!1,canUpload:!1},se={isRequested:!1,selectedUser:-1,selectedUserOpen:!1,results:[]},le={theme:"LIGHT"};function ce(){for(var e=[],t=0;t<arguments.length;t++)e[t]=arguments[t];return function(t,n){return e.reduce(function(e,t){return t(e,n)},t)}}function ue(){return Math.random().toString(36).substring(7)}function me(e){var t=e;"string"!==typeof e&&"number"!==typeof e||(t=new Date(e));var n=new Date,a=(n.getTime()-t.getTime())/1e3;return a<60?Math.floor(a)+" sec ago":a<3600?Math.floor(a/60)+" min ago":a<=86400?Math.floor(a/3600)+" hours ago":t.getDate()+" "+t.toDateString().match(/ [a-zA-Z]*/)[0].replace(" ","")+(t.getFullYear()===n.getFullYear()?"":" "+t.getFullYear())}var de=function(e){return"accuracy"===e?"Accuracy":"f1"===e?"F1 / Accuracy":"ps"===e?"Pearson-Spearman Corr":"matthews"===e?"Matthew's Corr":"Accuracy"},pe=function(e){return"-"===e.toString()?"-":(Math.round(1e3*e)/10).toFixed(1)},he=function(e){var t=["C","N","E"];return e.map(function(e){return e.map(function(e){return"<td>"+e+"</td>"}).join("")}).map(function(e,n){return"<th>"+t[n]+"</th>"+e}).join("\n")},fe=function(){return(fe=Object.assign||function(e){for(var t,n=1,a=arguments.length;n<a;n++)for(var r in t=arguments[n])Object.prototype.hasOwnProperty.call(t,r)&&(e[r]=t[r]);return e}).apply(this,arguments)};var ge=ce(function(e,t){return void 0===e&&(e=re),t.type===l?x.currentUser?fe(fe({},e),{isAuthenticated:!0,uid:x.currentUser.uid,avatarImageUrl:t.data.user.publicFields.avatarImageUrl,displayName:t.data.user.publicFields.displayName,isAdmin:!!t.data.user.isAdmin,isBanned:!!t.data.user.isBanned,email:t.data.user.privateFields.email,redirectToReferrer:!0,isCheckingLogin:!1}):e:t.type===c?fe(fe({},e),{isAuthenticated:!1,uid:null,isCheckingLogin:!1}):e},function(e,t){if(void 0===e&&(e=re),t.type===u){if(t.data.result||x.currentUser){var n=t.data.result||x.currentUser;return n?fe(fe({},e),{isAuthenticated:!0,uid:n.uid,redirectToReferrer:!0,isCheckingLogin:!1}):e}return e}return e},function(e,t){return void 0===e&&(e=re),t.type===s?x.currentUser?e:re:e},function(e,t){return void 0===e&&(e=re),t.type===y?fe(fe({},e),{isFetchingSubmissions:!0}):e},function(e,t){return void 0===e&&(e=re),t.type===v?fe(fe({},e),{isFetchingSubmissions:!1,submissions:t.data}):e},function(e,t){return void 0===e&&(e=re),t.type===w?fe(fe({},e),{isFetchingSubmissions:!0}):e},function(e,t){if(void 0===e&&(e=re),t.type===E){var n=fe(fe({},e),{isFetchingSubmissions:!1});return n.otherSubmissions[t.data.id]=t.data.data,n}return e}),be=ce(function(e,t){return void 0===e&&(e=oe),t.type===m?fe(fe({},e),{isRequested:!0}):e},function(e,t){return void 0===e&&(e=oe),t.type===d?fe(fe({},e),{isRequested:!1,tasks:t.data.result}):e}),ye=ce(function(e,t){return void 0===e&&(e=ie),t.type===f?fe(fe({},e),{isChecking:!0}):e},function(e,t){return void 0===e&&(e=ie),t.type===g?{isChecking:!1,
1canUpload:t.data}:e}),ve=ce(function(e,t){return void 0===e&&(e=se),t.type===p?fe(fe({},e),{isRequested:!0}):e},function(e,t){return void 0===e&&(e=se),t.type===h?fe(fe({},e),{isRequested:!1,results:t.data.result}):e},function(e,t){return void 0===e&&(e=se),t.type===b?fe(fe({},e),{selectedUserOpen:!e.selectedUserOpen,selectedUser:t.data}):e}),we=ce(function(e,t){return void 0===e&&(e=le),t.type===k?{theme:"LIGHT"===e.theme?"DARK":"LIGHT"}:e}),Ee=Object(ne.c)({userInfo:ge,taskList:be,submission:ye,resultList:ve,general:we,form:ae.c}),ke=n(197),Te=n.n(ke),Se=n(198),Ie=Object(Se.createLogger)(),Le=Object(ne.d)(Ee,Object(ne.a)(Te.a,Ie)),_e=function(){return(_e=Object.assign||function(e){for(var t,n=1,a=arguments.length;n<a;n++)for(var r in t=arguments[n])Object.prototype.hasOwnProperty.call(t,r)&&(e[r]=t[r]);return e}).apply(this,arguments)},xe=function(e,t){var n={};for(var a in e)Object.prototype.hasOwnProperty.call(e,a)&&t.indexOf(a)<0&&(n[a]=e[a]);if(null!=e&&"function"===typeof Object.getOwnPropertySymbols){var r=0;for(a=Object.getOwnPropertySymbols(e);r<a.length;r++)t.indexOf(a[r])<0&&Object.prototype.propertyIsEnumerable.call(e,a[r])&&(n[a[r]]=e[a[r]])}return n},Ne=function(e){var t,n=e.component,r=xe(e,["component"]);return a.createElement(o.e,_e({},r,{render:(t=n,function(e){if(!t)return null;if(Le.getState().userInfo.isAuthenticated)return a.createElement(t,_e({},e));var n={to:{pathname:"/login",state:{from:e.location}}};return a.createElement(o.d,_e({},n))})}))},Ce=function(){return(Ce=Object.assign||function(e){for(var t,n=1,a=arguments.length;n<a;n++)for(var r in t=arguments[n])Object.prototype.hasOwnProperty.call(t,r)&&(e[r]=t[r]);return e}).apply(this,arguments)},Oe=function(e,t){var n={};for(var a in e)Object.prototype.hasOwnProperty.call(e,a)&&t.indexOf(a)<0&&(n[a]=e[a]);if(null!=e&&"function"===typeof Object.getOwnPropertySymbols){var r=0;for(a=Object.getOwnPropertySymbols(e);r<a.length;r++)t.indexOf(a[r])<0&&Object.prototype.propertyIsEnumerable.call(e,a[r])&&(n[a[r]]=e[a[r]])}return n},Ae=function(e){var t,n=e.component,r=Oe(e,["component"]);return a.createElement(o.e,Ce({},r,{render:(t=n,function(e){if(!t)return null;if(Le.getState().userInfo.isAdmin)return a.createElement(t,Ce({},e));var n={to:{pathname:"/",state:{from:e.location}}};return a.createElement(o.d,Ce({},n))})}))},Re=n(28),Pe=n(27),Ue=n(56),De=function(){return(De=Object.assign||function(e){for(var t,n=1,a=arguments.length;n<a;n++)for(var r in t=arguments[n])Object.prototype.hasOwnProperty.call(t,r)&&(e[r]=t[r]);return e}).apply(this,arguments)},je=Object(Pe.createMuiTheme)({palette:{primary:{light:"#2b5283",main:"#2b5283",dark:"#2b5283",contrastText:"#fff"},secondary:{light:"#bd492a",main:"#e65f57",dark:"#e65f57",contrastText:"#000"}},typography:{useNextVariants:!0}}),Me={textAlign:"center",fontSize:"1em"},Be={textAlign:"center",fontWeight:"bold"},Ge=(De(De({},Be),{border:"1px solid #494647"}),{marginRight:"10px",cursor:"pointer"}),qe={display:"flex",alignItems:"center",justifyContent:"center"},Fe={textAlign:"center",textOverflow:"inherit"},We={whiteSpace:"pre",lineHeight:"1.2"},He=(De(De({},Fe),{textAlign:"left"}),De(De({},Fe),{textAlign:"right"}),{overflow:"hidden"}),ze=function(e){var t,n,a,r,o,i;return{fontIcon:(t={marginRight:"10px",cursor:"pointer",textAlign:"center",color:e.palette.primary.contrastText},t[e.breakpoints.down("md")]={color:e.palette.primary.main},t),superglueIcon:(n={height:"1.15em",verticalAlign:"sub"},n[e.breakpoints.down("md")]={height:"1em"},n),glueIcon:(a={height:"1.15em",verticalAlign:"sub",paddingRight:"0.2em"},a[e.breakpoints.down("md")]={height:"1em"},a),link:{display:"flex",alignItems:"center",justifyContent:"space-between",marginRight:"10px",color:e.palette.primary.contrastText,fontSize:"0.8em","&:hover":{textDecoration:"none",color:e.palette.secondary.main}},sectionDesktop:(r={display:"none"},r[e.breakpoints.up("lg")]={display:"flex"},r),sectionMobile:(o={display:"flex"},o[e.breakpoints.up("lg")]={display:"none"},o),grow:{flexGrow:1,justifyContent:"flex-start"},buttonLink:(i={color:e.palette.primary.contrastText,textDecoration:"none",marginRight:"0.9em","&:hover":{color:e.palette.secondary.main},"&:hover .icon-white":{display:"none"},"&:hover .icon-red":{display:"inline-block"}},i[e.breakpoints.down("md")]={fontSize:"0.8em",marginRight:"0.4em"},i)}},Qe=function(e){var t;return{form:(t={width:"40%",margin:"0 auto",textAlign:"center"},t[e.breakpoints.down("md")]={width:"90%"},t)}},Ve=function(e){return{th:De(De({},Be),{backgroundColor:e.palette.primary.main,color:e.palette.primary.contrastText}),fontIcon:De(De({},Ge),{color:e.palette.primary.main,borderBottom:"1px dotted "+e.palette.primary.main}),successIcon:De(De({},Ge),{color:"green"}),errorIcon:De(De({},Ge),{color:e.palette.secondary.main}),fontIconSidebar:De(De({},Ge),{color:e.palette.primary.main,fontSize:"1em",borderBottom:"1px dotted "+e.palette.primary.main}),table:{borderRadius:"4px"}
1,tr:De(De({borderBottom:"1px solid "+e.palette.secondary.contrastText},Me),{"& a:hover":{borderBottom:"1px solid transparent"},"& a":{borderBottom:"1px solid transparent"}}),textTr:De(De({borderBottom:"1px solid "+e.palette.secondary.contrastText},Me),{textAlign:"left","& a:hover":{borderBottom:"1px solid transparent"},"& a":{borderBottom:"1px solid transparent"}}),plusTr:De(De({borderBottom:"1px solid "+e.palette.secondary.contrastText},Me),{textAlign:"left",width:"0.2em",paddingLeft:"15px","& a:hover":{borderBottom:"1px solid transparent"},"& a":{borderBottom:"1px solid transparent"},"& span":{borderBottom:"1px solid transparent"}}),numberTr:De(De({borderBottom:"1px solid "+e.palette.secondary.contrastText},Me),{textAlign:"right","& a:hover":{borderBottom:"1px solid transparent"},"& a":{borderBottom:"1px solid transparent"}}),textTh:De(De({},Be),{backgroundColor:e.palette.primary.main,color:e.palette.primary.contrastText,textAlign:"left"}),plusTh:De(De({},Be),{backgroundColor:e.palette.primary.main,color:e.palette.primary.contrastText,textAlign:"left",width:"0.2em",paddingLeft:"15px"}),numberTh:De(De({},Be),{backgroundColor:e.palette.primary.main,color:e.palette.primary.contrastText,textAlign:"right"}),listItem:{display:"block"},listItemText:{display:"inline-block"}}},Je=function(e){var t;return{paper:(t={width:"15%"},t[e.breakpoints.down("sm")]={width:"80%"},t[e.breakpoints.down("md")]={width:"50%"},t)}},Ye=function(){return a.createElement("div",null,"We're cooking. Website should be fully up by tomorrow")},Ke=function(){var e=function(t,n){return(e=Object.setPrototypeOf||{__proto__:[]}instanceof Array&&function(e,t){e.__proto__=t}||function(e,t){for(var n in t)t.hasOwnProperty(n)&&(e[n]=t[n])})(t,n)};return function(t,n){function a(){this.constructor=t}e(t,n),t.prototype=null===n?Object.create(n):(a.prototype=n.prototype,new a)}}(),Xe=function(){return(Xe=Object.assign||function(e){for(var t,n=1,a=arguments.length;n<a;n++)for(var r in t=arguments[n])Object.prototype.hasOwnProperty.call(t,r)&&(e[r]=t[r]);return e}).apply(this,arguments)},Ze=n(50),$e=function(e){function t(t){var n=e.call(this,t)||this;return n.state={drawerOpen:!1,submission:null,lastDay:!1},n.submissions=[],n.getLastDayResults=function(e){n.submissions=[],n.setState({lastDay:e.currentTarget.checked||!1}),n.props.fetchResults(e.currentTarget.checked||!1)},n.handleCollapseClick=function(e,t){n.props.updateSelectedUser(t),e.stopPropagation()},n.handleCellClick=function(e,t){n.changeDrawerSubmission(n.submissions[e])},n.changeDrawerSubmission=function(e){n.setState({drawerOpen:!0,submission:e})},n.toggleDrawer=function(e){n.setState({drawerOpen:e})},n.getHeader=function(e,t){void 0===t&&(t=!0);var r=n.props.taskList.tasks;return a.createElement(Re.u,null,a.createElement(Re.v,null,a.createElement(Re.t,{className:n.props.classes.plusTh}),a.createElement(Re.t,{className:n.props.classes.numberTh},"Rank"),a.createElement(Re.t,{className:n.props.classes.textTh},"Name"),a.createElement(Re.t,{className:n.props.classes.textTh},"Model"),a.createElement(Re.t,{className:n.props.classes.th},"URL"),a.createElement(Re.t,{className:"task-score-column "+n.props.classes.numberTh},"Score"),Object.keys(r).filter(function(a){var r=n.props.taskList.tasks[a];return!r.type&&"primary"===e||("primary"===e&&!0===t&&"auxiliary"===r.type&&"main"===r.auxiliaryType||r.type===e&&("auxiliary"!==r.type||("subcategory"!==r.auxiliaryType||!0!==t)))}).map(function(e,t){return a.createElement(Re.t,{className:"task-score-column "+n.props.classes.numberTh,"data-tip":r[e].name+"-"+de(r[e].metric),key:ue()},r[e].identifier)})))},n.getRowColumn=function(e,t){if("ps"===t)return e.debug.split("/").map(function(e){return pe(parseFloat(e))}).join("/");if("undefined"===typeof e.score)return"-";if("-"===e.score.toString())return"-";var n=pe(e.score);if(0===e.score&&(n="0.0"),"f1"===t.toLowerCase()){var a=pe(e.correct/e.total);return n.toString()+"/"+a.toString()}return n},n.shouldDisplayStyle=function(e,t,a){return void 0===t&&(t=!1),void 0===a&&(a=!0),t||a||n.props.resultList.selectedUser===e&&n.props.resultList.selectedUserOpen?{}:{display:"none"}},n.getRow=function(e,t,r,o,i,s){return void 0===s&&(s=!0),n.submissions.push(r),a.createElement(Re.v,{style:n.shouldDisplayStyle(i,o,"GLUE Baselines"===t.displayName),key:ue(),onClick:function(){return n.changeDrawerSubmission(r)}},a.createElement(Re.t,{className:n.props.classes.plusTr},o&&"GLUE Baselines"!==t.displayName&&t.otherSubmissions&&t.otherSubmissions.length>0?n.props.resultList.selectedUserOpen&&n.props.resultList.selectedUser===i?a.createElement("div",{style:qe,onClick:function(e){return n.handleCollapseClick(e,i)}},a.createElement(Re.f,{className:"fa fa-minus "+n.props.classes.fontIcon})):a.createElement("div",{style:qe,onClick:function(e){return n.handleCollapseClick(e,i)}},a.createElement(Re.f,{className:"fa fa-plus "+n.props.classes.fontIcon})):""),a.createElement(Re.t,{className:n.props.classes.tr},o?i+1:""),a.createElement(Re.t,{className:n.props.classes.textTr},a.createElement(Ue,{effect:"solid",place:"right"}),o?t.displayName:""),a.createElement(Re.t,{"data-tip":r.editable?r.editable.name:"",className:n.props.classes.textTr},a.createElement(Ue,{effect:"solid",place:"right"}),r.editable?r.editable.name:""),a.createElement(Re.t,{className:n.props.classes.tr},r.editable&&r.editable.url&&r.editable.url.length?a.createElement("a",{href:r.editable.url},a.createElement(Re.f,{className:"fa fa-external-link "+n.props.classes.fontIcon})):""),a.createElement(Re.t,{className:"task-score-column "+n.props.classes.numberTr},pe(r.macroScore||0)),Object.keys(n.props.taskList.tasks).filter(function(t){var a=n.props.taskList.tasks[t];return!a.type&&"primary"===e||("primary"===e&&!0===s&&"auxiliary"===a.type&&"main"===a.auxiliaryType||a.type===e&&("auxiliary"!==a.type||"subcategory"!==a.auxiliaryType))}).map(function(e,t){var o=r.tasks[e]||{},i=n.props.taskList.tasks[e].metric,s=n.getRowColumn(o,i);return"QNLI"===n.props.taskList.tasks[e].identifier&&parseInt(r.createdOn,10)<1549027413e3&&(s="-"),a.createElement(Re.t,{className:"task-score-column "+n.props.classes.numberTr,key:ue(),"data-tip":s},s)}))},n.getTable=function(e,t){if(void 0===e&&(e="primary"),void 0===t&&(t=!0),0===n.props.resultList.results.length)return a.createElement("div",null);var r=n.getHeader(e,!0),o=a.createElement(Re.s,null,n.props.resultList.results.map(function(a,r){"GLUE Baselines"!==a.displayName&&(a.otherSubmissions=[]);var o=n.getRow(e,a,a.maxScoreSubmission,!0,r),i=a.otherSubmissions.map(function(o,i){return n.getRow(e,a,o,!1,r,t)});return i.unshift(o),i}));return a.createElement(Re.r,null,r,o)},n.getTableTabs=function(){return a.createElement("div",null,a.createElement(Re.p,{autoHideDuration:3e4,open:!0,message:"Click on a submission to see more information",style:{height:"auto",lineHeight:"28px",padding:12,whiteSpace:"pre-line"}}),n.getTable("primary",!0),a.createElement("br",null),a.createElement("br",null))},n.getDiagnostics=function(e){if(!e)return{};var t={};
1return Object.keys(e.tasks).forEach(function(a){var r=n.props.taskList.tasks[a],o=Xe(Xe({},e.tasks[a]),{name:r.name});if("auxiliary"===r.type)if("main"===r.auxiliaryType)t[r.identifier]||(t[r.identifier]={}),t[r.identifier].main=o;else if("category"===r.auxiliaryType){var i=r.identifier.substr(0,2);t[i]||(t[i]={}),t[i].categories||(t[i].categories={}),t[i].categories[r.identifier]||(t[i].categories[r.identifier]={}),t[i].categories[r.identifier].main=o}else if("subcategory"===r.auxiliaryType){i=r.identifier.substr(0,2);t[i]||(t[i]={}),t[i].categories||(t[i].categories={});var s=r.identifier.substr(0,4);t[i].categories[s]||(t[i].categories[s]={}),t[i].categories[s].categories||(t[i].categories[s].categories={}),t[i].categories[s].categories[r.identifier]=o}}),t},n.getDiagnosticsList=function(e){return Object.keys(e).map(function(t){var r=e[t],o=he(JSON.parse(r.main.confusion)),i=a.createElement(Re.k,{className:n.props.classes.listItem},a.createElement(Re.l,null,r.main.name," confusion matrix"),a.createElement(Re.l,{className:n.props.classes.listItemText},a.createElement("table",{className:"diagnostics-table"},a.createElement("thead",null,a.createElement("tr",null,a.createElement("th",
1null),a.createElement("th",null,"C"),a.createElement("th",null,"N"),a.createElement("th",null,"E"))),a.createElement("tbody",null,o.split("\n").map(function(e){return a.createElement("tr",{key:ue(),dangerouslySetInnerHTML:{__html:e}})})))),a.createElement(Re.l,null,"C = contradiction"),a.createElement(Re.l,null,"N = not entailment"),a.createElement(Re.l,null,"E = entailment"));r.categories||(r.categories={});var s=Object.keys(r.categories).map(function(e){var t=r.categories[e];t.categories||(t.categories={});var o=Object.keys(t.categories).map(function(e){var r=t.categories[e],o=r.name.substr(t.main.name.length).trim().replace(/-/g,"");return a.createElement(Re.k,{className:n.props.classes.listItem,key:ue()},o+": "+pe(r.score))});return a.createElement(Re.k,{className:n.props.classes.listItem,key:ue()},t.main.name+": "+pe(t.main.score),o)});return a.createElement("div",{key:ue()},i,a.createElement(Re.c,null),a.createElement("b",null,a.createElement(Re.k,null,"Category-wise Matthew's Correlation Scores")),s,a.createElement(Re.c,null))})},n.getDrawerContent=function(){var e=n.state.submission;if(!e)return a.createElement("div",null);var t=n.getDiagnostics(e);return a.createElement(Re.j,{style:{textAlign:"left"}},a.createElement("b",null,a.createElement(Re.k,null,e.editable.name)),a.createElement(Re.k,null,"GitHub/Model URL: \xa0",e.editable.url?a.createElement("a",{href:e.editable.url},a.createElement(Re.f,{className:"fa fa-external-link "+n.props.classes.fontIconSidebar})):"N/A"),a.createElement(Re.k,null,"Submitted: ",me(e.createdOn)),a.createElement(Re.k,null,"Score: ",a.createElement("b",null,"\xa0",pe(e.macroScore))),a.createElement(Re.k,{style:{color:T.THEME[n.props.general.theme].BLUE}},a.createElement(o.b,{to:"/submission/"+e.uploaderUid+"/"+e.submissionId},"More details \xa0",a.createElement(Re.f,{className:"fa fa-external-link "+n.props.classes.fontIconSidebar}))),a.createElement(Re.c,null),a.createElement("b",null,a.createElement(Re.k,null,"Model Description")),a.createElement(Re.k,null,e.editable.modelDescription),a.createElement(Re.c,null),a.createElement("b",null,a.createElement(Re.k,null,"Parameter Description")),a.createElement(Re.k,null,e.editable.parameterDescription),a.createElement(Re.c,null),a.createElement(Re.k,null,a.createElement("b",null,"Total Parameters:"),"\xa0 ",e.editable.totalParameters),a.createElement(Re.k,null,a.createElement("b",null,"Shared Parameters:"),"\xa0 ",e.editable.sharedParameters),a.createElement(Re.c,null),a.createElement(Re.k,null,a.createElement("b",null,"Diagnostics Information")),n.getDiagnosticsList(t))},n.submissions=[],n}return Ke(t,e),t.prototype.componentDidMount=function(){this.props.taskList.tasks&&0!==Object.keys(this.props.taskList.tasks).length||this.props.fetchTasks(),this.props.fetchResults(this.state.lastDay)},t.prototype.render=function(){var e=this;if(!this.props.userInfo.isSiteReady)return a.createElement(Ye,null);var t=Object(Pe.withStyles)(Je)(Re.d);return a.createElement("div",null,this.props.resultList.isRequested||this.props.taskList.isRequested?a.createElement("div",{style:He},a.createElement("p",null,"Fetching leaderboard..."),a.createElement("img",{src:Ze,className:"App-logo",alt:"Loading"})):a.createElement("div",null,a.createElement(t,{anchor:"right",className:"leaderboard-drawer",open:this.state.drawerOpen,onClose:function(){return e.toggleDrawer(!1)}},this.getDrawerContent()),this.props.userInfo.isAdmin?a.createElement("label",null,a.createElement("input",{name:"last_day",type:"checkbox",checked:this.state.lastDay,onClick:this.getLastDayResults}),"Recent top 50"):"",this.getTableTabs()))},t}(a.Component),et=Object(Pe.withStyles)(Ve)($e),tt=Object(i.b)(function(e){return{userInfo:e.userInfo,resultList:e.resultList,taskList:e.taskList,general:e.general}},function(e){return{fetchTasks:function(){return e(M())},fetchResults:function(t){return e(W(t))},updateSelectedUser:function(t){return e(function(e){return{type:b,data:e}}(t))}}})(et),nt=n(39),at=n.n(nt),rt=n(96),ot=n.n(rt),it=n(54),st=n.n(it),lt=n(46),ct=n.n(lt),ut=n(95),mt=n.n(ut),dt=n(97),pt=n.n(dt),ht=n(199),ft=n.n(ht),gt=n(200),bt=n.n(gt),yt=function(){var e=function(t,n){return(e=Object.setPrototypeOf||{__proto__:[]}instanceof Array&&function(e,t){e.__proto__=t}||function(e,t){for(var n in t)t.hasOwnProperty(n)&&(e[n]=t[n])})(t,n)};return function(t,n){function a(){this.constructor=t}e(t,n),t.prototype=null===n?Object.create(n):(a.prototype=n.prototype,new a)}}(),vt=n(424).faqs,wt=function(e){function t(){var t=null!==e&&e.apply(this,arguments)||this;return t.state={status:Array.apply(null,Array(vt.length)).map(function(){return 0})},t.toggle=function(e){var n=t.state.status.slice();n[e]=Number(!n[e]),t.setState({status:n})},t.getStyles=function(e){return t.state.status[e]?{}:{display:"none"}},t}return yt(t,e),t.prototype.render=function(){var e=this;return a.createElement("div",{className:"faq"},a.createElement(at.a,{elevation:0,style:{backgroundColor:"#ececec"}},a.createElement(st.a,null,a.createElement(ot.a,{disableSticky:!0},"Frequently asked questions"),vt.length>0?vt.map(function(t,n){return a.createElement(st.a,{key:ue()},a.createElement(ct.a,{className:e.props.classes.listItem,onClick:function(){return e.toggle(n)}},a.createElement(mt.a,{primary:n+1+". "+t.question}),1===e.state.status[n]?a.createElement(ft.a,null):a.createElement(bt.a,null)),a.createElement(pt.a,{in:1===e.state.status[n],key:n},a.createElement(st.a,null,a.createElement(ct.a,{style:{lineHeight:1.4},key:ue()},a.createElement("div",{dangerouslySetInnerHTML:{__html:Array.isArray(t.answer)?t.answer.join(" "):t.answer}}
1)))))}):"")))},t}(a.Component),Et=Object(Pe.withStyles)(function(e){return{listItem:{cursor:"pointer"}}})(wt),kt=function(){var e=function(t,n){return(e=Object.setPrototypeOf||{__proto__:[]}instanceof Array&&function(e,t){e.__proto__=t}||function(e,t){for(var n in t)t.hasOwnProperty(n)&&(e[n]=t[n])})(t,n)};return function(t,n){function a(){this.constructor=t}e(t,n),t.prototype=null===n?Object.create(n):(a.prototype=n.prototype,new a)}}(),Tt=n(50),St={name:"Name",identifier:"Identifier",downloadURL:"Download",infoURL:"More Info",metric:"Metric"},It=function(e){function t(){var t=null!==e&&e.apply(this,arguments)||this;return t.getTasksHeader=function(){return a.createElement(Re.u,null,a.createElement(Re.v,null,Object.keys(St).map(function(e,n){return a.createElement(Re.t,{colSpan:"name"===e?2:1,className:t.props.classes.th,key:e+n},St[e])})))},t.getRowColumn=function(e,n){return"downloadURL"===n?"auxiliary"===e.type&&"main"!==e.auxiliaryType?"":a.createElement("a",{href:e[n]},a.createElement(X.a,{className:"fa fa-download "+t.props.classes.fontIcon})):"infoURL"===n?"auxiliary"===e.type&&"main"!==e.auxiliaryType?"":a.createElement("a",{href:e[n]},a.createElement(X.a,{className:"fa fa-external-link "+t.props.classes.fontIcon})):"metric"===n?e[n]?de(e[n]):"":e[n]||""},t.getTaskRow=function(e,n){return a.createElement(Re.v,{key:n},Object.keys(St).map(function(r,o){return a.createElement(Re.t,{colSpan:"name"===r?2:1,key:n+o,className:t.props.classes.tr,"data-tip":"name"===r?t.getRowColumn(e,r):""},a.createElement(Ue,{effect:"solid",place:"bottom"}),t.getRowColumn(e,r))}))},t.getTasksTable=function(e){if(void 0===e&&(e="primary"),!t.props.taskList.tasks||0===Object.keys(t.props.taskList.tasks).length)return a.createElement("div",null);var n=Object.keys(t.props.taskList.tasks),r=t.getTasksHeader(),o=a.createElement(Re.s,null,n.filter(function(n){var a=t.props.taskList.tasks[n].type,r=t.props.taskList.tasks[n].identifier;return!a&&"primary"===e||("AX"===r||a===e)}).map(function(e,n){return t.getTaskRow(t.props.taskList.tasks[e],e)}));return a.createElement(Re.r,{className:t.props.classes.table},r,o)},t.getTableTabs=function(){return t.getTasksTable()},t}return kt(t,e),t.prototype.componentDidMount=function(){this.props.fetchTaskList()},t.prototype.render=function(){return this.props.userInfo.isSiteReady?this.props.taskList.isRequested?a.createElement("div",{style:He},a.createElement("p",null,"Fetching tasks..."),a.createElement("img",{src:Tt,className:"App-logo",alt:"Loading"})):a.createElement("div",{className:"tasks-container"},a.createElement("h2",null,a.createElement("strong",null,"GLUE Tasks")),this.getTableTabs(),a.createElement("br",null),a.createElement("a",{title:"Download Script",href:"https://github.com/nyu-mll/jiant/blob/master/scripts/download_glue_data.py"},a.createElement(Y.a,{color:"primary",variant:"contained"},"Download data"),a.createElement("br",null)),a.createElement("br",null),this.props.userInfo.isAuthenticated&&this.props.userInfo.isAdmin?a.createElement(o.b,{to:"/addtask"},a.createElement(Y.a,{color:"primary",variant:"contained"},"Add Task")):"",a.createElement("br",null)):a.createElement(Ye,null)},t}(a.Component),Lt=Object(Pe.withStyles)(Ve)(It),_t=Object(i.b)(function(e){return{taskList:e.taskList,userInfo:e.userInfo,general:e.general}},function(e){return{fetchTaskList:function(){return e(M())}}})(Lt),xt=function(){return window.location.href="https://super.gluebenchmark.com",a.createElement("div",{className:"App-intro"},a.createElement(V.a,{to:"https://super.gluebenchmark.com"}))},Nt=(n(427),n(32)),Ct=n.n(Nt),Ot=n(201),At=n.n(Ot),Rt=function(){var e=function(t,n){return(e=Object.setPrototypeOf||{__proto__:[]}instanceof Array&&function(e,t){e.__proto__=t}||function(e,t){for(var n in t)t.hasOwnProperty(n)&&(e[n]=t[n])})(t,n)};return function(t,n){function a(){this.constructor=t}e(t,n),t.prototype=null===n?Object.create(n):(a.prototype=n.prototype,new a)}}(),Pt=function(){return(Pt=Object.assign||function(e){for(var t,n=1,a=arguments.length;n<a;n++)for(var r in t=arguments[n])Object.prototype.hasOwnProperty.call(t,r)&&(e[r]=t[r]);return e}).apply(this,arguments)},Ut=Object(Pe.withStyles)(ze)(function(e){return a.createElement(Re.m,{anchorEl:e.mobileMoreAnchorEl,anchorOrigin:{vertical:"top",horizontal:"right"},transformOrigin:{vertical:"top",horizontal:"right"},open:e.isMobileMenuOpen,onClose:e.handleMenuClose},a.createElement(Re.b,{onClickAway:e.handleMenuClose},a.createElement(Re.n,{onClick:e.handleMenuClose},a.createElement(Re.i,{href:"https://openreview.net/pdf?id=rJ4km2R5t7",className:e.classes.link},a.createElement(Ct.a,null,a.createElement(Re.f,{className:"fa fa-file "+e.classes.fontIcon}),"Paper")),a.createElement(Re.i,{href:"https://github.com/nyu-mll/GLUE-baselines",className:e.classes.link},a.createElement(Ct.a,null,a.createElement(Re.f,{className:"fa fa-code "+e.classes.fontIcon}),"Code")),a.createElement(o.c,{activeClassName:"active-link",to:"/tasks",className:e.classes.link},a.createElement(Ct.a,null,a.createElement(Re.f,{className:"fa fa-tasks "+e.classes.fontIcon}),"Tasks")),a.createElement(o.c,{activeClassName:"active-link",to:"/leaderboard",className:e.classes.link},a.createElement(Ct.a,null,a.createElement(Re.f,{className:"fa fa-trophy "+e.classes.fontIcon}),"Leaderboard")),a.createElement(o.c,{activeClassName:"active-link",to:"/faq",className:e.classes.link},a.createElement(Ct.a,null,a.createElement(Re.f,{className:"fa fa-info "+e.classes.fontIcon}),"FAQ")),a.createElement(o.c,{activeClassName:"active-link",to:"/diagnostics",className:e.classes.link},a.createElement(Ct.a,null,a.createElement(Re.f,{className:"fa fa-bug "+e.classes.fontIcon}),"Diagnostics")),a.createElement(o.c,{activeClassName:"active-link",to:"/submit",className:e.classes.link},a.createElement(Ct.a,null,a.createElement(Re.f,{className:"fa fa-paper-plane "+e.classes.fontIcon}),"Submit")),e.userInfo.isAuthenticated?a.createElement(o.c,{activeClassName:"active-link",to:"/profile/"+e.userInfo.uid,className:e.classes.link},a.createElement(Ct.a,null,a.createElement(Re.f,{className:"fa fa-user "+e.classes.fontIcon}),"Profile")):"",e.userInfo.isAuthenticated?a.createElement("div",{className:e.classes.link,onClick:function(t){return e.onLogout()}},a.createElement(Ct.a,null,a.createElement(Re.f,{className:"fa fa-sign-out "+e.classes.fontIcon}),"Logout")):a.createElement(o.c,{activeClassName:"active-link",to:"/login",className:e.classes.link},a.createElement(Ct.a,null,a.createElement(Re.f,{className:"fa fa-sign-in "+e.classes.fontIcon}),"Login")))))}),Dt=function(e){function t(){var t=null!==e&&e.apply(this,arguments)||this;return t.state={mobileMoreAnchorEl:null},t.handleMobileMenuOpen=function(e){t.setState({mobileMoreAnchorEl:e.currentTarget})},t.handleMobileMenuClose=function(){t.setState({mobileMoreAnchorEl:null})},t}return Rt(t,e),t.prototype.render=function(){var e=this,t=this.state.mobileMoreAnchorEl,n=Boolean(t);return a.createElement("div",{className:"App-title"},a.createElement(Re.a,{position:"static",color:"primary"},a.createElement(Re.y,null,a.createElement("div",{className:this.props.classes.grow},a.createElement(Re.i,{underline:"none",align:"left",className:this.props.classes.buttonLink+" active-link",href:"/"},a.createElement(Re.f,null,a.createElement("img",{className:this.props.classes.glueIcon,title:"GLUE",alt:"GLUE icon",src:"/assets/images/glue_icon_red.svg"})),"GLUE"),a.createElement(Re.i,{underline:"none",align:"left",className:this.props.classes.buttonLink,href:"https://super.gluebenchmark.com/"},a.createElement(Re.f,null,a.createElement("img",{className:this.props.classes.superglueIcon+" icon-white",title:"SuperGLUE",alt:"SuperGLUE icon",src:"/assets/images/superglue_icon_white.svg"}),a.createElement("img",{className:this.props.classes.superglueIcon+" icon-red",title:"SuperGLUE",alt:"SuperGLUE icon",src:"/assets/images/superglue_icon_red.svg"}))," SuperGLUE")),a.createElement("div",{className:this.props.classes.sectionDesktop},a.createElement(Re.i,{href:"https://openreview.net/pdf?id=rJ4km2R5t7",className:this.props.classes.link},a.createElement(Re.f,{className:"fa fa-file "+this.props.classes.fontIcon}),"Paper"),a.createElement(Re.i,{href:"https://github.com/nyu-mll/GLUE-baselines",className:this.props.classes.link},a.createElement(Re.f,{className:"fa fa-code "+this.props.classes.fontIcon}),"Code"),a.createElement(o.c,{activeClassName:"active-link",to:"/tasks",className:this.props.classes.link},a.createElement(Re.f,{className:"fa fa-tasks "+this.props.classes.fontIcon}),"Tasks"),a.createElement(o.c,{activeClassName:"active-link",to:"/leaderboard",className:this.props.classes.link},a.createElement(Re.f,{className:"fa fa-trophy "+this.props.classes.fontIcon}),"Leaderboard"),a.createElement(o.c,{activeClassName:"active-link",to:"/faq",className:this.props.classes.link},a.createElement(Re.f,{className:"fa fa-info "+this.props.classes.fontIcon}),"FAQ"),a.createElement(o.c,{activeClassName:"active-link",to:"/diagnostics",className:this.props.classes.link},a.createElement(Re.f,{className:"fa fa-bug "+this.props.classes.fontIcon}),"Diagnostics"),a.createElement(o.c,{activeClassName:"active-link",to:"/submit",className:this.props.classes.link},a.createElement(Re.f,{className:"fa fa-paper-plane "+this.props.classes.fontIcon}),"Submit"),this.props.userInfo.isAuthenticated?a.createElement(o.c,{activeClassName:"active-link",to:"/profile/"+this.props.userInfo.uid,className:this.props.classes.link},a.createElement(Re.f,{className:"fa fa-user "+this.props.classes.fontIcon}),"Profile"):"",this.props.userInfo.isAuthenticated?a.createElement(Re.i,{className:this.props.classes.link,onClick:function(t){return e.props.onLogout()}},a.createElement(Re.f,{className:"fa fa-sign-out "+this.props.classes.fontIcon}),"Logout"):a.createElement(o.c,{activeClassName:"active-link",to:"/login",className:this.props.classes.link},a.createElement(Re.f,{className:"fa fa-sign-in "+this.props.classes.fontIcon}),"Login")),a.createElement("div",{className:this.props.classes.sectionMobile},a.createElement(Re.g,{"aria-haspopup":"true",onClick:this.handleMobileMenuOpen,color:"inherit"},a.createElement(At.a,null))),a.createElement(Ut,Pt({mobileMoreAnchorEl:t,isMobileMenuOpen:n,handleMenuClose:this.handleMobileMenuClose},this.props)))))},t}(a.Component),jt=Object(Pe.withStyles)(ze)(Dt),Mt=n(34),Bt=n.n(Mt),Gt=n(430),qt=n(431),Ft=n(432),Wt=Object(Pe.withStyles)(function(e){var t,n,a;return{logo:{width:"100%",borderBottom:"1px dotted transparent","&:hover":{textDecoration:"none",border:"none"}},firstLogo:(t={marginLeft:"10%"},t[e.breakpoints.down("sm")]={marginLeft:0},t),buttonLinks:{"& a":{borderBottom:"1px dotted transparent"}},logoContainer:{justifyContent:"center",alignItems:"center",alignContent:"center",paddingTop:"2em"},logoImage:{width:"100%"},glueName:(n={textAlign:"left",color:e.palette.primary.main,fontSize:"3.5em"},n[e.breakpoints.up("sm")]={fontSize:"4em"},n[e.breakpoints.up("md")]={fontSize:"4.8em"},n[e.breakpoints.up("lg")]={fontSize:"6em"},n),glueLogoContainer:{textAlign:"right"},glueLogo:(a={color:e.palette.primary.main,width:"4.75em"},a[e.breakpoints.up("sm")]={width:"5.75em"},a[e.breakpoints.up("md")]={width:"6.75em"},a[e.breakpoints.up("lg")]={width:"7.75em"},a)}})(function(e){return a.createElement("div",{className:"App-intro"},a.createElement(at.a,{elevation:0,style:{backgroundColor:"transparent"}},a.createElement(Bt.a,{container:!0,className:"App-intro-heading",alignContent:"space-around",justify:"space-around"},a.createElement(Bt.a,{item:!0,xs:1,sm:3}),a.createElement(Bt.a,{className:e.classes.glueLogoContainer,item:!0,xs:3,sm:2},a.createElement("img",{className:e.classes.glueLogo,title:"GLUE",alt:"GLUE logo",src:"assets/images/glue_icon_blue.svg"})),a.createElement(Bt.a,{className:e.classes.glueName,item:!0,xs:8,sm:4},"GLUE"),a.createElement(Bt.a,{item:!0,xs:1,sm:3})),a.createElement("br",null),a.createElement(Bt.a,{container:!0,className:e.classes.logoContainer},a.createElement(Bt.a,{item:!0,xs:12,sm:4,md:4,lg:4},a.createElement(Bt.a,{container:!0,className:[e.classes.logoContainer,e.classes.firstLogo].join(" ")},a.createElement(Bt.a,{item:!0,xs:10,sm:8,md:8},a.createElement("a",{className:e.classes.logo,href:"https://wp.nyu.edu/ml2/",title:"ML^2 NYU"},a.createElement("img",{alt:"ML^2 NYU",className:e.classes.logoImage,title:"ML^2 NYU Logo",src:Gt}))))),a.createElement(Bt.a,{item:!0,xs:12,sm:3,md:3,lg:3},a.createElement(Bt.a,{container:!0,className:e.classes.logoContainer},a.createElement(Bt.a,{item:!0,xs:10,sm:10,md:10},a.createElement("a",{className:e.classes.logo,href:"https://www.cs.washington.edu/research/nlp",title:"UW NLP"},a.createElement("img",{alt:"UW NLP",className:e.classes.logoImage,title:"UW NLP Logo",src:qt}))))),a.createElement(Bt.a,{item:!0,xs:12,sm:4,md:4,lg:4},a.createElement(Bt.a,{container:!0,className:e.classes.logoContainer},a.createElement(Bt.a,{item:!0,xs:10,sm:9,md:9},a.createElement("a",{className:e.classes.logo,href:"https://deepmind.com/",title:"DeepMind"},a.createElement("img",{alt:"DeepMind",className:e.classes.logoImage,title:"DeepMind Logo",src:Ft}))))),a.createElement(Bt.a,{item:!0,xs:12,md:1,lg:1})),a.createElement("div",{className:"App-intro-content"},"The General Language Understanding Evaluation (GLUE) benchmark is a collection of resources for training, evaluating, and analyzing natural language understanding systems. GLUE consists of:",a.createElement("ul",null,a.createElement("li",null,"A benchmark of nine sentence- or sentence-pair language understanding tasks built on established existing datasets and selected to cover a diverse range of dataset sizes, text genres, and degrees of difficulty,"),a.createElement("li",null,"A diagnostic dataset designed to evaluate and analyze model performance with respect to a wide range of linguistic phenomena found in natural language, and"),a.createElement("li",null,"A public leaderboard for tracking performance on the benchmark and a dashboard for visualizing the performance of models on the diagnostic set.")),"The format of the GLUE benchmark is model-agnostic, so any system capable of processing sentence and sentence pairs and producing corresponding predictions is eligible to participate. The benchmark tasks are selected so as to favor models that share information across tasks using parameter sharing or other transfer learning techniques. The ultimate goal of GLUE is to drive research in the development of general and robust natural language understanding systems.",a.createElement("br",null),a.createElement("br",null)),a.createElement(Bt.a,{className:e.classes.buttonLinks,container:!0,alignContent:"center",justify:"center"},a.createElement("a",{href:"https://openreview.net/pdf?id=rJ4km2R5t7",title:"Paper"},a.createElement(Y.a,{variant:"contained",color:"primary",style:{marginBottom:"1em"}},"Paper")),"\xa0",a.createElement("a",{href:"https://github.com/nyu-mll/GLUE-baselines",title:"Starter Code"},a.createElement(Y.a,{variant:"contained",color:"primary",style:{marginBottom:"1em"}},"Starter Code")),"\xa0",a.createElement("a",{href:"https://groups.google.com/forum/#!forum/glue-benchmark-discuss",title:"Google Group"},a.createElement(Y.a,{variant:"contained",color:"primary",style:{marginBottom:"1em"}},"Group")),"\xa0",a.createElement(o.b,{to:"/diagnostics"},a.createElement(Y.a,{variant:"contained",color:"primary",style:{marginBottom:"1em"}},"Diagnostics"))),a.createElement("br",null),a.createElement("br",null)),a.createElement("br",null),a.createElement("br",null))}),Ht=n(202),zt=n(203),Qt=function(){var e=new Ht.Converter;return a.createElement("div",{className:"diagnostic-page"},a.createElement("h1",{style:{textAlign:"center"}},"GLUE Diagnostic Dataset"),a.createElement("div",{dangerouslySetInnerHTML:{__html:e.makeHtml(zt)}}))},Vt=function(){return(Vt=Object.assign||function(e){for(var t,n=1,a=arguments.length;n<a;n++)for(var r in t=arguments[n])Object.prototype.hasOwnProperty.call(t,r)&&(e[r]=t[r]);return e}).apply(this,arguments)},Jt=function(e){return function(t){return e(t.target.files[0])}},Yt=function(e){var t=e.input,n=t.value;return a.createElement(Y.a,{variant:"contained",color:"primary",component:"label",onClick:function(e){}},n?n.name:e.label,a.createElement("input",Vt({style:{display:"none"},onChange:Jt(t.onChange),onBlur:Jt(t.onBlur),type:"file"},e)))},Kt=function(){var e=function(t,n)
1{return(e=Object.setPrototypeOf||{__proto__:[]}instanceof Array&&function(e,t){e.__proto__=t}||function(e,t){for(var n in t)t.hasOwnProperty(n)&&(e[n]=t[n])})(t,n)};return function(t,n){function a(){this.constructor=t}e(t,n),t.prototype=null===n?Object.create(n):(a.prototype=n.prototype,new a)}}(),Xt=function(e){function t(){var t=null!==e&&e.apply(this,arguments)||this;return t.state={value:""},t}return Kt(t,e),t.prototype.render=function(){var e=this,t=this.props;return a.createElement(Re.e,{fullWidth:!0},a.createElement(Re.h,null,t.label),a.createElement(Re.o,{value:0!==this.state.value.length?this.state.value:t.input.value,onChange:function(n){t.input.onChange(n.target.value),e.setState({value:n.target.value})}},t.children))},t}(a.Component),Zt=function(e){function t(){var t=null!==e&&e.apply(this,arguments)||this;return t.state={value:""},t}return Kt(t,e),t.prototype.render=function(){var e=this,t=this.props;return a.createElement(Re.e,{fullWidth:!0},a.createElement(Re.x,{onChange:function(n){t.input.onChange(n.target.value),e.setState({value:n.target.value})},multiline:!!t.multiline&&t.multiline,value:0!==this.state.value.length?this.state.value:t.input.value,label:t.label,inputProps:{onFocus:t.focus,onBlur:t.blur}}))},t}(a.Component),$t=function(e){var t=e.handleSubmit,n=e.reset;return e.submitSucceeded?a.createElement("div",null,a.createElement("span",null,"Upload Successful"),a.createElement("br",null),a.createElement(Y.a,{variant:"contained",color:"primary",onClick:n},"Reset")):a.createElement("form",{className:"centered-container",onSubmit:t,style:{width:"30%",margin:"0 auto"}},a.createElement(ae.a,{name:"identifier",fullWidth:!0,component:Zt,label:"Task Identifier"}),a.createElement("br",null),a.createElement(ae.a,{name:"name",fullWidth:!0,component:Zt,label:"Task Name"}),a.createElement("br",null),a.createElement(ae.a,{name:"metric",component:Xt,label:"Metric"},a.createElement(Ct.a,{value:"accuracy"},"Accuracy"),a.createElement(Ct.a,{value:"f1"},"F1"),a.createElement(Ct.a,{value:"ps"},"Pearson and Spearman"),a.createElement(Ct.a,{value:"matthews"},"Matthews"),a.createElement(Ct.a,{value:"multirc"},"MultiRC"),a.createElement(Ct.a,{value:"record"},"ReCoRD"),a.createElement(Ct.a,{value:"winogender"},"Winogender")),a.createElement("br",null),a.createElement(ae.a,{name:"weight",fullWidth:!0,component:Zt,type:"number",label:"Weight"}),a.createElement("br",null),a.createElement(ae.a,{name:"type",component:Xt,label:"Task Type"},a.createElement(Ct.a,{value:"primary"},"Primary"),a.createElement(Ct.a,{value:"auxiliary"},"Auxiliary")),a.createElement("br",null),a.createElement(ae.a,{name:"auxiliaryType",component:Xt,label:"Auxiliary Task Type"},a.createElement(Ct.a,{value:"main"},"Main"),a.createElement(Ct.a,{value:"category"},"Category"),a.createElement(Ct.a,{value:"subcategory"},"SubCategory")),a.createElement("br",null),a.createElement(ae.a,{name:"downloadURL",fullWidth:!0,component:Zt,label:"Download URL"}),a.createElement(ae.a,{name:"infoURL",fullWidth:!0,component:Zt,label:"Info URL"}),a.createElement("br",null),a.createElement("br",null),a.createElement(ae.a,{type:"file",name:"testTSV",component:Yt,label:T.SELECT_TSV_LABEL,accept:".tsv"}),a.createElement("br",null),a.createElement("br",null),a.createElement("br",null),a.createElement(Y.a,{variant:"contained",color:"primary",type:"submit"},"Add Task"))},en=$t=Object(ae.d)({form:"addTask"})($t),tn=function(){var e=function(t,n){return(e=Object.setPrototypeOf||{__proto__:[]}instanceof Array&&function(e,t){e.__proto__=t}||function(e,t){for(var n in t)t.hasOwnProperty(n)&&(e[n]=t[n])})(t,n)};return function(t,n){function a(){this.constructor=t}e(t,n),t.prototype=null===n?Object.create(n):(a.prototype=n.prototype,new a)}}(),nn=function(e){function t(){var t=null!==e&&e.apply(this,arguments)||this;return t.submit=function(e){if(!t.props.userInfo.isAdmin)throw new ae.b({_error:"You need to be admin"});["identifier","name","metric","type","testTSV","downloadURL"].forEach(function(t){if(!e[t])throw new ae.b({_error:"Missing "+t})}),e.weight=parseFloat(e.weight);var n=_.ref("tasksAdminInfo"),a=n.push().key;return t.uploadTaskTSV(e,a).then(function(){return t.addTaskTSVToDB(n,a).then(function(){return t.addTaskToDB(e,a).then(function(){return console.log("Uploaded")},function(e){return t.throwError("Add to DB failed",e)})},function(e){return t.throwError("Add to DB failed",e)})},function(e){return t.throwError("Upload failed",e)})},t.addTaskTSVToDB=function(e,t){return e.child(t).set({tsvPath:"tasksTSV/"+t+".tsv"})},t.addTaskToDB=function(e,t){delete e.testTSV;var n=_.ref("tasks");return e.tsvWithoutLabels="https://not_uploaded_yet",n.child(t).set(e)},t.throwError=function(e,t){throw void 0===e&&(e="Failed"),new ae.b({message:e,_error:t})},t.isValidSubmission=function(){return t.props.userInfo.isAuthenticated&&t.props.userInfo.isAdmin},t.uploadTaskTSV=function(e,n){return new Promise(function(a,r){t.isValidSubmission()||r("You need to be an admin to upload");var o=N.ref().child("tasksTSV"),i=n+".tsv",s=o.child(i),l={customMetadata:{uploaderUid:t.props.userInfo.uid,originalName:e.testTSV.name,uploaderEmail:t.props.userInfo.email}};s.put(e.testTSV,l).then(function(e){a(e)},function(){r("TSV upload failed")})})},t}return tn(t,e),t.prototype.render=function(){return a.createElement(en,{initialValues:{weight:1},onSubmit:this.submit})},t}(a.Component),an=Object(i.b)(function(e){return{userInfo:e.userInfo}})(nn),rn=n(98),on=n.n(rn),sn=function(){var e=function(t,n){return(e=Object.setPrototypeOf||{__proto__:[]}instanceof Array&&function(e,t){e.__proto__=t}||function(e,t){for(var n in t)t.hasOwnProperty(n)&&(e[n]=t[n])})(t,n)};return function(t,n){function a(){this.constructor=t}e(t,n),t.prototype=null===n?Object.create(n):(a.prototype=n.prototype,new a)}}(),ln=n(50),cn=function(e){function t(){var t=null!==e&&e.apply(this,arguments)||this;return t.state={open:"none",anchorEl:null},t.messages={name:"A short name for your system, which will be displayed in the leaderboard. (Required)",url:"A URL for a paper or (if one is not available) website or \ncode repository describing your system.",modelDescription:"A sentence or tw
1o describing your system. Make sure to mention any outside\ndata or resources you use. (Required)",parameterDescription:"A sentence or explaining how you share parameters across tasks\n(or stating that you don't share parameters). (Required)",totalParameters:"The total number of trained parameters in your model.\nDo not count word or word-part embedding parameters, even if they are trained.",sharedParameters:"The total number of trained parameters in your model that are shared across\nmultiple task models. If some parameters are shared across some but not all\nof your tasks, count those. Do not count word or word-part embedding parameters,\neven if they are trained.",public:"Whether you want your submission to be visible on the public leaderboard."},t.handleClick=function(e,n){t.setState({open:n,anchorEl:e.currentTarget})},t.handleRequestClose=function(){t.setState({open:"none",anchorEl:null})},t}return sn(t,e),t.prototype.render=function(){var e=this,t=this.props,n=t.handleSubmit,r=t.submitSucceeded,o=t.submitting,i=!!this.props.isEditForm,s=this.handleClick.bind(this);return o?a.createElement("div",{className:this.props.classes.form},a.createElement("p",null,"Submitting..."),a.createElement("img",{src:ln,className:"App-logo",alt:"Loading"})):r?a.createElement("div",{className:this.props.classes.form},a.createElement("span",null,i?"Update":"Submit"," Successful"),a.createElement("br",null),a.createElement(Y.a,{variant:"contained",color:"primary",onClick:function(){window.location.reload()}},"Reset")):a.createElement("form",{onSubmit:n,className:this.props.classes.form},i?"":"Please make sure that you are submitting your results with newer version of QNLI.",a.createElement("br",null),a.createElement(on.a,{open:Boolean(this.state.anchorEl),placement:"right-start",anchorEl:this.state.anchorEl},a.createElement(at.a,null,a.createElement(ct.a,{style:We},this.messages[this.state.open]))),a.createElement(ae.a,{name:"name",component:Zt,maxLength:50,label:"Submission Name*",focus:function(e){return s(e,"name")},blur:function(){return e.handleRequestClose()}}),a.createElement("br",null),a.createElement(ae.a,{name:"url",component:Zt,maxLength:200,label:"Model URL/Github",focus:function(e){return s(e,"url")},blur:function(){return e.handleRequestClose()}}),a.createElement("br",null),a.createElement(ae.a,{name:"modelDescription",multiline:!0,component:Zt,label:"Model Description*",rowsMax:4,maxLength:500,focus:function(e){return s(e,"modelDescription")},blur:function(){return e.handleRequestClose()}}),a.createElement("br",null),a.createElement(ae.a,{name:"parameterDescription",multiline:!0,component:Zt,rowsMax:4,maxLength:500,label:"Parameter sharing description*",focus:function(e){return s(e,"parameterDescription")},blur:function(){return e.handleRequestClose()}}),a.createElement("br",null),a.createElement(ae.a,{name:"totalParameters",maxLength:50,component:Zt,type:"number",label:"Total number of parameters",focus:function(e){return s(e,"totalParameters")},blur:function(){return e.handleRequestClose()}}),a.createElement("br",null),a.createElement(ae.a,{name:"sharedParameters",maxLength:50,component:Zt,type:"number",label:"Shared number of parameters",focus:function(e){return s(e,"sharedParameters")},blur:function(){return e.handleRequestClose()}}),a.createElement("br",null),a.createElement("br",null),a.createElement(ae.a,{name:"public",component:"input",type:"checkbox",label:"Public?"}),a.createElement("label",null," Public?"),i?"":a.createElement("div",null,a.createElement("br",null),a.createElement("br",null),a.createElement(ae.a,{type:"file",name:"file",component:Yt,label:T.SELECT_ZIP_LABEL,accept:".zip"})),a.createElement("br",null),a.createElement("br",null),a.createElement(Y.a,{variant:"contained",color:"primary",disabled:o,type:"submit"},i?"Update":"Submit"))},t}(a.Component),un=Object(ae.d)({form:"submitForm"})(Object(Pe.withStyles)(Qe)(cn)),mn=Object(Pe.withStyles)(Qe)(cn),dn=un,pn=function(){var e=function(t,n){return(e=Object.setPrototypeOf||{__proto__:[]}instanceof Array&&function(e,t){e.__proto__=t}||function(e,t){for(var n in t)t.hasOwnProperty(n)&&(e[n]=t[n])})(t,n)};return function(t,n){function a(){this.constructor=t}e(t,n),t.prototype=null===n?Object.create(n):(a.prototype=n.prototype,new a)}}
1(),hn=n(50),fn=function(e){function t(){var t=null!==e&&e.apply(this,arguments)||this;return t.submit=function(e){["name","modelDescription","parameterDescription","file"].forEach(function(t){if("undefined"===typeof e[t])throw new ae.b({_error:"Missing "+t})});var n=_.ref().child("submissions").push().key;if(!n)throw Object(ae.b)("Connection to DB failed");return t.uploadZip(e,n).then(function(){return t.addSubmissionToUser(n).then(function(){return t.addSubmissionToDB(e,n).then(function(){return console.log("Upload successful")},function(){return t.throwError("Add to DB failed")})},function(){return t.throwError("Add to DB failed")})},function(){return t.throwError("Upload failed")})},t.throwError=function(e){throw new ae.b({_error:e})},t.isValidSubmission=function(){return t.props.userInfo.isAuthenticated},t.addSubmissionToUser=function(e){var n={createdOn:S.a.database.ServerValue.TIMESTAMP};return _.ref().child("users/"+t.props.userInfo.uid).child("publicButAddable/submissions/"+e).set(n)},t.addSubmissionToDB=function(e,n){var a="submissions/"+n+".zip";"undefined"===typeof e.totalParameters&&(e.totalParameters=-1),"undefined"===typeof e.sharedParameters&&(e.sharedParameters=-1);var r={uploaderUid:t.props.userInfo.uid,filePath:a,name:e.name,url:e.url||"",public:!!e.public,modelDescription:e.modelDescription,parameterDescription:e.parameterDescription,totalParameters:parseInt(e.totalParameters,10),sharedParameters:parseInt(e.sharedParameters,10),createdOn:S.a.database.ServerValue.TIMESTAMP,isGraded:!1};return _.ref().child("submissions/"+n).set(r)},t.uploadZip=function(e,n){return new Promise(function(a,r){t.isValidSubmission()||r("You need to be authenticated to upload");var o=N.ref().child("submissions"),i=n+".zip",s=o.child(i),l={customMetadata:{uploaderUid:t.props.userInfo.uid,originalName:e.file.name,uploaderEmail:t.props.userInfo.email}};s.put(e.file,l).then(function(e){a(e)},function(){r("Zip upload failed")})})},t}return pn(t,e),t.prototype.componentDidMount=function(){this.props.doCanUploadCheck()},t.prototype.render=function(){return this.props.userInfo.isSiteReady?this.props.submission.isChecking?a.createElement("div",{className:"submission-container",style:He},a.createElement("p",null,"Checking whether you can upload"),a.createElement("img",{src:hn,className:"App-logo",alt:"Loading"})):a.createElement(dn,{onSubmit:this.submit}):a.createElement(Ye,null)},t}(a.Component),gn=Object(i.b)(function(e){return{userInfo:e.userInfo,submission:e.submission}},function(e){return{doCanUploadCheck:function(){return e(q())}}})(fn),bn=function(){var e=function(t,n){return(e=Object.setPrototypeOf||{__proto__:[]}instanceof Array&&function(e,t){e.__proto__=t}||function(e,t){for(var n in t)t.hasOwnProperty(n)&&(e[n]=t[n])})(t,n)};return function(t,n){function a(){this.constructor=t}e(t,n),t.prototype=null===n?Object.create(n):(a.prototype=n.prototype,new a)}}(),yn=n(50),vn=function(e){function t(){var t=null!==e&&e.apply(this,arguments)||this;return t.getHeader=function(){return a.createElement(Re.u,null,a.createElement(Re.v,null,a.createElement(Re.t,{className:t.props.classes.th},"Model"),a.createElement(Re.t,{className:t.props.classes.th},"Status"),a.createElement(Re.t,{className:t.props.classes.th},"URL"),a.createElement(Re.t,{className:t.props.classes.th},"Score"),a.createElement(Re.t,{className:t.props.classes.th},"Created On")))},t.getRows=function(){return a.createElement(Re.s,null,Object.keys(t.props.userInfo.submissions).map(function(e,n){var r=t.props.userInfo.submissions[e],i="undefined"===typeof r.successful||r.successful;return a.createElement(Re.v,{key:e},a.createElement(Re.t,{className:t.props.classes.tr},a.createElement(o.b,{to:"/submission/"+t.props.userInfo.uid+"/"+e},r.editable&&r.editable.name||"")),a.createElement(Re.t,{className:t.props.classes.tr},a.createElement(Ue,{effect:"solid",html:!0,place:"right","data-multiline":!0}),i?a.createElement(Re.f,{className:"fa fa-check "+t.props.classes.successIcon}):a.createElement(Re.f,{"data-tip":t.getFormattedMessage(r.message),className:"fa fa-exclamation-triangle "+t.props.classes.errorIcon})),a.createElement(Re.t,{className:t.props.classes.tr},r.editable&&r.editable.url&&r.editable.url.length>0?a.createElement("a",{href:r.editable.url},a.createElement(Re.f,{className:"fa fa-external-link "+t.props.classes.fontIcon})):""),a.createElement(Re.t,{className:t.props.classes.tr},i?pe(r.macroScore):"Failed"),a.createElement(Re.t,{className:t.props.classes.tr},me(r.createdOn)))}))},t.getFormattedMessage=function(e){return e||(e=""),e.split("\n").join("<br/>")},t.getSubmissionsTable=function(){if(t.props.userInfo.isFetchingSubmissions)return a.createElement("div",null,a.createElement("p",null,"Fetching your submissions..."),a.createElement("img",{src:yn,className:"App-logo",alt:"Loading"}));if(0===Object.keys(t.props.userInfo.submissions).length)return a.createElement("div",null,"No submissions as of now. Keep crunching!");var e=t.getHeader(),n=t.getRows();return a.createElement(Re.r,null,e,n)},t}return bn(t,e),t.prototype.componentDidMount=function(){if(this.props.match&&this.props.match.params.uid){var e=this.props.match.params.uid;this.props.fetchUserSubmissions(e)}},t.prototype.render=function(){return a.createElement("div",{className:"profile-container"},a.createElement("h1",null,this.props.userInfo.displayName,"'s submissions"),a.createElement("p",null,"(Hover over \xa0",a.createElement(Re.f,{style:{verticalAlign:"text-bottom"},className:"fa fa-exclamation-triangle "+this.props.classes.errorIcon}
1),"for more details on failures.)"),this.getSubmissionsTable(),a.createElement("br",null))},t}(a.Component),wn=Object(Pe.withStyles)(Ve)(vn),En=Object(i.b)(function(e){return{userInfo:e.userInfo,general:e.general}},function(e){return{fetchUserSubmissions:function(t){return e(H(t))}}})(wn),kn=function(){var e=function(t,n){return(e=Object.setPrototypeOf||{__proto__:[]}instanceof Array&&function(e,t){e.__proto__=t}||function(e,t){for(var n in t)t.hasOwnProperty(n)&&(e[n]=t[n])})(t,n)};return function(t,n){function a(){this.constructor=t}e(t,n),t.prototype=null===n?Object.create(n):(a.prototype=n.prototype,new a)}}(),Tn=function(){return(Tn=Object.assign||function(e){for(var t,n=1,a=arguments.length;n<a;n++)for(var r in t=arguments[n])Object.prototype.hasOwnProperty.call(t,r)&&(e[r]=t[r]);return e}).apply(this,arguments)},Sn=function(e){return a.createElement(mn,{userInfo:e.userInfo,isEditForm:!0,handleSubmit:e.handleSubmit,submitSucceeded:e.submitSucceeded,submitting:e.submitting})};Sn=Object(ae.d)({form:"submissionEditForm"})(Object(Pe.withStyles)(Qe)(Sn));var In=function(e){function t(){var t=null!==e&&e.apply(this,arguments)||this;return t.submit=function(e){["name","url","modelDescription","parameterDescription","public"].forEach(function(t){if("undefined"===typeof e[t])throw new ae.b({_error:"Missing "+t})}),"undefined"===typeof e.totalParameters&&(e.totalParameters=-1),"undefined"===typeof e.sharedParameters&&(e.sharedParameters=-1);var n=t.props.submissionId,a=e.public,r={name:e.name,modelDescription:e.modelDescription,url:e.url,parameterDescription:e.parameterDescription,public:!!e.public,totalParameters:parseInt(e.totalParameters,10),sharedParameters:parseInt(e.sharedParameters,10)},o=_.ref().child("resultsVisibility/"+n),i=_.ref().child("results/"+n+"/editable");return Promise.all([o.set({public:a}),i.set(r)]).then(function(){return console.log("Upload successful")},function(e){return t.throwError("Upload failed"+e)})},t.throwError=function(e){throw new ae.b({_error:e})},t}return kn(t,e),t.prototype.render=function(){var e=this.props.submission,t=Tn({},e.editable);return t.public=!!t.public,a.createElement(Sn,{initialValues:t,onSubmit:this.submit})},t}(a.Component),Ln=Object(i.b)(function(e){return{userInfo:e.userInfo}},function(e){return{}})(In),_n=function(){var e=function(t,n){return(e=Object.setPrototypeOf||{__proto__:[]}instanceof Array&&function(e,t){e.__proto__=t}||function(e,t){for(var n in t)t.hasOwnProperty(n)&&(e[n]=t[n])})(t,n)};return function(t,n){function a(){this.constructor=t}e(t,n),t.prototype=null===n?Object.create(n):(a.prototype=n.prototype,new a)}}(),xn=function(){return(xn=Object.assign||function(e){for(var t,n=1,a=arguments.length;n<a;n++)for(var r in t=arguments[n])Object.prototype.hasOwnProperty.call(t,r)&&(e[r]=t[r]);return e}).apply(this,arguments)},Nn=n(50),Cn=function(e){function t(){var t=null!==e&&e.apply(this,arguments)||this;return t.state={selectedTab:0},t.changeTab=function(e,n){t.setState({selectedTab:n})},t.getTasks=function(e,n){return void 0===n&&(n=!1),Object.keys(t.props.taskList.tasks).filter(function(a){return t.props.taskList.tasks[a].type===e&&(!n||"AX"===t.props.taskList.tasks[a].identifier)}).map(function(e){var n=t.props.taskList.tasks[e];return[e,n.name,n.identifier,n.metric]})},t.getRowColumn=function(e,t){if("ps"===t)return e.debug.split("/").map(function(e){return pe(parseFloat(e))}).join("/");if("undefined"===typeof e)return"-";if("undefined"===typeof e.score)return"-";if("-"===e.score.toString())return"-";var n=pe(e.score);if(0===e.score&&(n="0.0"),"f1"===t.toLowerCase()){var a=pe(e.correct/e.total);return n.toString()+"/"+a.toString()}return n},t.getRows=function(e,n,r){return void 0===r&&(r=!1),t.getTasks(n,r).map(function(n,r){var o=t.props.taskList.tasks[n[0]].identifier,i=t.getRowColumn(e.tasks[n[0]],n[3]);return"QNLI"===o&&parseInt(e.createdOn,10)<1549027413e3&&(i="-"),a.createElement(Re.v,{key:ue()},a.createElement(Re.t,{colSpan:2,className:t.props.classes.tr},n[1]),a.createElement(Re.t,{className:t.props.classes.tr},n[2]),a.createElement(Re.t,{className:t.props.classes.tr},de(n[3])),a.createElement(Re.t,{className:t.props.classes.tr},i))})},t.displayTable=function(e,n){return t.state.selectedTab===n?a.createElement(Re.r,null,a.createElement(Re.u,null,a.createElement(Re.v,null,a.createElement(Re.t,{colSpan:2,className:t.props.classes.th},"Task"),a.createElement(Re.t,{className:t.props.classes.th},"Task"),a.createElement(Re.t,{className:t.props.classes.th},"Metric"),a.createElement(Re.t,{className:t.props.classes.th},"Score"))),a.createElement(Re.s,null,e)):""},t.a11yProps=function(e){return{id:"tab-"+e,"aria-controls":"tabpanel-"+e}}
1,t.displaySubmission=function(e){var n=t.getRows(e,"primary"),r=t.getRows(e,"auxiliary",!0);n=n.concat(r);var o=t.getRows(e,"auxiliary");return a.createElement("div",{className:"submission-container"},t.props.userInfo.uid===t.props.match.params.uid||t.props.userInfo.isAdmin?a.createElement(Ln,{submission:e,submissionId:t.props.match.params.submissionId}):"",a.createElement("h2",null,"Results for submission ",a.createElement("strong",null,e.editable.name)),a.createElement("h3",null,"Score: ",a.createElement("strong",null,pe(e.macroScore))),a.createElement(Re.w,{centered:!0,value:t.state.selectedTab,onChange:t.changeTab},a.createElement(Re.q,xn({value:0,label:"Primary"},t.a11yProps(0))),a.createElement(Re.q,xn({value:1,label:"Diagnostics"},t.a11yProps(1)))),t.displayTable(n,0),t.displayTable(o,1),a.createElement("br",null),a.createElement("br",null))},t}return _n(t,e),t.prototype.componentDidMount=function(){var e=this.props.match.params.submissionId;this.props.fetchSubmission(e),0===Object.keys(this.props.taskList.tasks).length&&this.props.fetchTaskList()},t.prototype.render=function(){var e=this.props.match.params.submissionId,t=!1,n="No submission found.";return this.props.userInfo.otherSubmissions[e]&&(t=!0,!1===this.props.userInfo.otherSubmissions[e].successful&&(t=!1,n="This submission was unsuccessful.")),this.props.userInfo.isFetchingSubmissions||this.props.taskList.isRequested?a.createElement("div",{style:He},a.createElement("p",null,"Checking for submission..."),a.createElement("img",{src:Nn,className:"App-logo",alt:"Loading"})):t?this.displaySubmission(this.props.userInfo.otherSubmissions[e]):a.createElement("div",null,n)},t}(a.Component),On=Object(Pe.withStyles)(Ve)(Cn),An=Object(i.b)(function(e){return{userInfo:e.userInfo,taskList:e.taskList,general:e.general}},function(e){return{fetchTaskList:function(){return e(M())},fetchSubmission:function(t){return e(z(t))}}})(On),Rn=function(e){var t=(e.location.state||{from:"/"}).from;return t&&e.location.pathname!==t?e.userInfo.redirectToReferrer&&a.createElement(V.a,{to:t||"/"}):a.createElement("div",null)},Pn=function(){var e=function(t,n){return(e=Object.setPrototypeOf||{__proto__:[]}instanceof Array&&function(e,t){e.__proto__=t}||function(e,t){for(var n in t)t.hasOwnProperty(n)&&(e[n]=t[n])})(t,n)};return function(t,n){function a(){this.constructor=t}e(t,n),t.prototype=null===n?Object.create(n):(a.prototype=n.prototype,new a)}}(),Un=function(){return(Un=Object.assign||function(e){for(var t,n=1,a=arguments.length;n<a;n++)for(var r in t=arguments[n])Object.prototype.hasOwnProperty.call(t,r)&&(e[r]=t[r]);return e}).apply(this,arguments)},Dn=function(e){function t(){return null!==e&&e.apply(this,arguments)||this}return Pn(t,e),t.prototype.componentDidMount=function(){var e=this;x.onAuthStateChanged(function(t){e.props.onAuthChanged(t)})},t.prototype.render=function(){var e,t,n=this;return e=this.props.general.theme,t=T.THEME[e],document.body.style.backgroundColor=t.BODY_BACKGROUND,document.body.style.color=t.YELLOW,a.createElement(Pe.MuiThemeProvider,{theme:je},a.createElement("div",{className:"App"},a.createElement("header",{className:"App-header"},a.createElement(jt,{history:this.props.history,userInfo:this.props.userInfo,onLogout:this.props.onLogout,general:this.props.general})),a.createElement("div",{className:"App-container"},a.createElement(V.b,{exact:!0,path:"/",render:function(e){return a.createElement(Rn,Un({},e,{userInfo:n.props.userInfo}))}}),a.createElement(V.b,{exact:!0,path:"/",component:Wt}),a.createElement(Ne,{path:"/submit",component:gn}),a.createElement("div",{className:"detail-container"},a.createElement(V.b,{path:"/diagnostics",component:Qt}),a.createElement(V.b,{path:"/faq",component:Et})),a.createElement(Ae,{path:"/addtask",component:an}),a.createElement(V.b,{path:"/login",component:te})),a.createElement("div",{className:"leaderboard-container"},a.createElement(V.b,{path:"/leaderboard",component:tt})),a.createElement("div",{className:"App-container"},a.createElement(V.b,{path:"/super",component:xt})),a.createElement("div",{className:"other-container"},a.createElement(Ne,{path:"/profile/:uid",component:En}),a.createElement(V.b,{path:"/tasks",component:_t}),a.createElement(V.b,{path:"/submission/:uid/:submissionId",component:An}))))},t}(a.Component),jn=Object(i.b)(function(e){return{userInfo:e.userInfo,general:e.general}},function(e){return{onAuthChanged:function(t){return e(U(t))},onLogout:function(){return e(j())}}})(Dn),Mn=n(142),Bn=Boolean("localhost"===window.location.hostname||"[::1]"===window.location.hostname||window.location.hostname.match(/^127(?:\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3}$/));function Gn(e){navigator.serviceWorker.register(e).then(function(e){e.onupdatefound=function(){var t=e.installing;t&&(t.onstatechange=function(){"installed"===t.state&&(navigator.serviceWorker.controller?console.log("New content is available; please refresh."):console.log("Content is cached for offline use."))})}}).catch(function(e){console.error("Error during service worker registration:",e)})}n(437);r.render(a.createElement(function(){return a.createElement(i.a,{store:Le}
1,a.createElement(o.a,{basename:Mn.basename||"/"},a.createElement("div",null,a.createElement(o.e,{path:"/",component:jn}))))},null),document.getElementById("root")),function(){if("serviceWorker"in navigator){if(new URL("",window.location.toString()).origin!==window.location.origin)return;window.addEventListener("load",function(){var e="/service-worker.js";Bn?function(e){fetch(e).then(function(t){404===t.status||-1===t.headers.get("content-type").indexOf("javascript")?navigator.serviceWorker.ready.then(function(e){e.unregister().then(function(){window.location.reload()})}):Gn(e)}).catch(function(){console.log("No internet connection found. App is running in offline mode.")})}(e):Gn(e)})}}()},50:function(e,t,n){e.exports=n.p+"static/media/loading.cc606db7.svg"}},[[260,2,1]]]); 2//# sourceMappingURL=main.5af841fb.chunk.js.map
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.